Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
2026-11-18T02:54:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-18T02:54:41Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task8fD3U4/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task8fD3U4/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/security.properties
2026-11-18 02:54:48: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task8fD3U4/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task8fD3U4/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20261118 02:54:56.114955 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.126:38025
--webserver_interface=127.2.229.126
--webserver_port=0
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 02:54:56.452553 3246 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:54:56.452838 3246 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:54:56.452914 3246 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:54:56.460963 3246 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:54:56.461047 3246 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:54:56.461100 3246 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:54:56.461146 3246 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:54:56.471426 3246 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.126:38025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.229.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:54:56.473129 3246 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:54:56.474905 3246 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:54:56.484417 3252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:56.484465 3251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:56.484555 3254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:54:56.485175 3246 server_base.cc:1061] running on GCE node
I20261118 02:54:56.485896 3246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:54:56.487358 3246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:54:56.488569 3246 hybrid_clock.cc:648] HybridClock initialized: now 1794970496488476 us; error 79 us; skew 500 ppm
I20261118 02:54:56.488901 3246 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:54:56.490999 3246 webserver.cc:577] Webserver started at http://127.2.229.126:44955/ using document root <none> and password file <none>
I20261118 02:54:56.491662 3246 fs_manager.cc:362] Metadata directory not provided
I20261118 02:54:56.491770 3246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:54:56.492172 3246 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 02:54:56.494459 3246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/instance:
uuid: "26c6b20fa203404a88463ea27bdd3c7d"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:54:56.495209 3246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal/instance:
uuid: "26c6b20fa203404a88463ea27bdd3c7d"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:54:56.500836 3246 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20261118 02:54:56.504983 3260 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:56.506525 3246 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20261118 02:54:56.506685 3246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
uuid: "26c6b20fa203404a88463ea27bdd3c7d"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:54:56.506874 3246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:54:56.532059 3246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:54:56.533010 3246 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:54:56.533267 3246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:54:56.552716 3246 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.126:38025
I20261118 02:54:56.552734 3311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.126:38025 every 8 connection(s)
I20261118 02:54:56.554250 3246 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/info.pb
I20261118 02:54:56.559012 3312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:54:56.561719 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 3246
I20261118 02:54:56.562021 2965 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal/instance
I20261118 02:54:56.565382 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.125:38339
--webserver_interface=127.2.229.125
--webserver_port=0
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 02:54:56.566968 3312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } has no permanent_uuid. Determining permanent_uuid...
I20261118 02:54:56.585426 3312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
W20261118 02:54:56.586551 3262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:54:56.588035 3312 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.125:38339: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:54:56.637395 3312 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } attempt: 1
W20261118 02:54:56.639678 3312 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.125:38339: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:54:56.707960 3312 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } attempt: 2
W20261118 02:54:56.710085 3312 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.125:38339: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:54:56.801340 3312 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } attempt: 3
W20261118 02:54:56.803478 3312 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.125:38339: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:54:56.902096 3313 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:54:56.902383 3313 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:54:56.902487 3313 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:54:56.911231 3313 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:54:56.911329 3313 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:54:56.911386 3313 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:54:56.911448 3313 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:54:56.922199 3313 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.125:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.229.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:54:56.923800 3313 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:54:56.925546 3313 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:54:56.933743 3321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:56.933895 3322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:56.934798 3324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:54:56.935501 3313 server_base.cc:1061] running on GCE node
I20261118 02:54:56.936355 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:54:56.937865 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:54:56.939128 3313 hybrid_clock.cc:648] HybridClock initialized: now 1794970496939055 us; error 58 us; skew 500 ppm
I20261118 02:54:56.939474 3313 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:54:56.941597 3313 webserver.cc:577] Webserver started at http://127.2.229.125:39093/ using document root <none> and password file <none>
I20261118 02:54:56.942291 3313 fs_manager.cc:362] Metadata directory not provided
I20261118 02:54:56.942401 3313 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:54:56.942763 3313 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 02:54:56.945137 3313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/instance:
uuid: "457eb38cba564ecd9f6f11245172260b"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:54:56.945937 3313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal/instance:
uuid: "457eb38cba564ecd9f6f11245172260b"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:54:56.946704 3312 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } attempt: 4
W20261118 02:54:56.948648 3312 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.125:38339: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:54:56.951696 3313 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20261118 02:54:56.955060 3330 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:56.956616 3313 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261118 02:54:56.956769 3313 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
uuid: "457eb38cba564ecd9f6f11245172260b"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:54:56.956960 3313 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:54:56.987924 3313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:54:56.988865 3313 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:54:56.989114 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:54:57.010638 3313 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.125:38339
I20261118 02:54:57.010656 3381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.125:38339 every 8 connection(s)
I20261118 02:54:57.012235 3313 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
I20261118 02:54:57.014909 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 3313
I20261118 02:54:57.015105 2965 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal/instance
I20261118 02:54:57.016510 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.124:36695
--webserver_interface=127.2.229.124
--webserver_port=0
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 02:54:57.017807 3382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:54:57.024376 3382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } has no permanent_uuid. Determining permanent_uuid...
I20261118 02:54:57.040587 3382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
I20261118 02:54:57.050233 3382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } has no permanent_uuid. Determining permanent_uuid...
W20261118 02:54:57.051327 3333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
W20261118 02:54:57.052856 3382 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:54:57.102182 3382 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } attempt: 1
W20261118 02:54:57.104190 3382 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:54:57.172462 3382 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } attempt: 2
W20261118 02:54:57.174477 3382 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:54:57.247923 3312 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } attempt: 5
I20261118 02:54:57.254782 3312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } has no permanent_uuid. Determining permanent_uuid...
W20261118 02:54:57.256673 3312 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:54:57.265739 3382 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } attempt: 3
W20261118 02:54:57.267822 3382 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:54:57.307953 3312 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } attempt: 1
W20261118 02:54:57.310200 3312 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
W20261118 02:54:57.361761 3383 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:54:57.362092 3383 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:54:57.362171 3383 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:54:57.370635 3383 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:54:57.370723 3383 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:54:57.370775 3383 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:54:57.370821 3383 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:54:57.378453 3312 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } attempt: 2
W20261118 02:54:57.380460 3312 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:54:57.381361 3383 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.124:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.229.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:54:57.382948 3383 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:54:57.384711 3383 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:54:57.393571 3391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:57.393539 3392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:57.393714 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:54:57.394506 3383 server_base.cc:1061] running on GCE node
I20261118 02:54:57.395190 3383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:54:57.396548 3383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:54:57.397734 3383 hybrid_clock.cc:648] HybridClock initialized: now 1794970497397681 us; error 43 us; skew 500 ppm
I20261118 02:54:57.398057 3383 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:54:57.400012 3383 webserver.cc:577] Webserver started at http://127.2.229.124:39557/ using document root <none> and password file <none>
I20261118 02:54:57.400686 3383 fs_manager.cc:362] Metadata directory not provided
I20261118 02:54:57.400769 3383 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:54:57.401073 3383 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 02:54:57.403333 3383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/instance:
uuid: "813a6937cea442ef81645fccc32207f4"
format_stamp: "Formatted at 2026-11-18 02:54:57 on dist-test-slave-49p1"
I20261118 02:54:57.404026 3383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal/instance:
uuid: "813a6937cea442ef81645fccc32207f4"
format_stamp: "Formatted at 2026-11-18 02:54:57 on dist-test-slave-49p1"
I20261118 02:54:57.409183 3383 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20261118 02:54:57.411052 3382 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } attempt: 4
I20261118 02:54:57.412738 3400 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20261118 02:54:57.413053 3382 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:54:57.414247 3383 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20261118 02:54:57.414533 3383 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
uuid: "813a6937cea442ef81645fccc32207f4"
format_stamp: "Formatted at 2026-11-18 02:54:57 on dist-test-slave-49p1"
I20261118 02:54:57.414742 3383 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:54:57.428887 3383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:54:57.429796 3383 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:54:57.430063 3383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:54:57.450080 3383 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.124:36695
I20261118 02:54:57.450137 3451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.124:36695 every 8 connection(s)
I20261118 02:54:57.451520 3383 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
I20261118 02:54:57.453918 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 3383
I20261118 02:54:57.454114 2965 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal/instance
I20261118 02:54:57.456431 3452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:54:57.463096 3452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } has no permanent_uuid. Determining permanent_uuid...
I20261118 02:54:57.475638 3452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
I20261118 02:54:57.483482 3452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } has no permanent_uuid. Determining permanent_uuid...
I20261118 02:54:57.486673 3312 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } attempt: 3
I20261118 02:54:57.497180 3452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap starting.
I20261118 02:54:57.498615 3312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Bootstrap starting.
I20261118 02:54:57.500439 3452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Neither blocks nor log segments found. Creating new log.
I20261118 02:54:57.501435 3452 log.cc:826] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Log is configured to *not* fsync() on all Append() calls
I20261118 02:54:57.501684 3312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Neither blocks nor log segments found. Creating new log.
I20261118 02:54:57.502955 3312 log.cc:826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Log is configured to *not* fsync() on all Append() calls
I20261118 02:54:57.504689 3452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: No bootstrap required, opened a new log
I20261118 02:54:57.505167 3312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: No bootstrap required, opened a new log
I20261118 02:54:57.510484 3452 raft_consensus.cc:360] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:57.510754 3452 raft_consensus.cc:386] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:54:57.510862 3452 raft_consensus.cc:748] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813a6937cea442ef81645fccc32207f4, State: Initialized, Role: FOLLOWER
I20261118 02:54:57.510764 3312 raft_consensus.cc:360] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:57.511034 3312 raft_consensus.cc:386] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:54:57.511140 3312 raft_consensus.cc:748] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c6b20fa203404a88463ea27bdd3c7d, State: Initialized, Role: FOLLOWER
I20261118 02:54:57.511459 3452 consensus_queue.cc:261] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:57.511744 3312 consensus_queue.cc:261] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:57.512408 3462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:57.512722 3462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:57.513617 3452 sys_catalog.cc:565] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:54:57.513945 3312 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:54:57.512709 3463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:57.514487 3463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
W20261118 02:54:57.530668 3487 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261118 02:54:57.530831 3487 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261118 02:54:57.526643 3246 master_runner.cc:187] Error getting master registration for 127.2.229.125:38339: OK, instance_id { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" instance_seqno: 1794970497004867 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20261118 02:54:57.534158 3246 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:54:57.535902 3266 tablet.cc:2441] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261118 02:54:57.536818 3493 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261118 02:54:57.536950 3493 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261118 02:54:57.537336 3383 master_runner.cc:187] Error getting master registration for 127.2.229.125:38339: OK, instance_id { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" instance_seqno: 1794970497004867 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20261118 02:54:57.543740 3383 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:54:57.601938 3463 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:54:57.602249 3463 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:57.604494 3463 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:54:57.605849 3427 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4" is_pre_election: true
I20261118 02:54:57.606558 3427 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 0.
I20261118 02:54:57.607404 3263 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4; no voters:
I20261118 02:54:57.608546 3463 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:54:57.608690 3463 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:54:57.608775 3463 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:54:57.612150 3463 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:57.623270 3427 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4"
I20261118 02:54:57.623477 3427 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:54:57.627024 3427 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 1.
I20261118 02:54:57.627271 3463 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 1 election: Requested vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:54:57.628038 3263 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4; no voters:
I20261118 02:54:57.628651 3463 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:54:57.652120 3463 raft_consensus.cc:705] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 1 LEADER]: Becoming Leader. State: Replica: 26c6b20fa203404a88463ea27bdd3c7d, State: Running, Role: LEADER
I20261118 02:54:57.653203 3463 consensus_queue.cc:238] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:57.675707 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 1 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:57.676425 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
W20261118 02:54:57.682088 3406 tablet.cc:2441] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261118 02:54:57.684000 3500 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 02:54:57.686430 3500 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 02:54:57.686980 3496 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:54:57.706310 3462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 1 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:57.706588 3462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:57.712242 3382 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } attempt: 5
I20261118 02:54:57.717248 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:57.717730 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
I20261118 02:54:57.730195 3382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap starting.
I20261118 02:54:57.770082 3382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Neither blocks nor log segments found. Creating new log.
I20261118 02:54:57.778677 3382 log.cc:826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Log is configured to *not* fsync() on all Append() calls
I20261118 02:54:57.786593 3462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:57.786809 3462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:57.807310 3382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: No bootstrap required, opened a new log
I20261118 02:54:57.810382 3500 catalog_manager.cc:1398] Generated new cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:54:57.810503 3500 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:54:57.836112 3500 catalog_manager.cc:1421] Generated new certificate authority record
I20261118 02:54:57.838007 3500 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:54:57.843020 3382 raft_consensus.cc:360] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:57.844609 3382 raft_consensus.cc:386] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:54:57.845155 3382 raft_consensus.cc:748] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457eb38cba564ecd9f6f11245172260b, State: Initialized, Role: FOLLOWER
I20261118 02:54:57.848489 3382 consensus_queue.cc:261] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:57.859978 3500 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Generated new TSK 0
I20261118 02:54:57.861334 3500 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 02:54:57.862920 3500 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:54:57.864259 3500 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 02:54:57.868654 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:57.868974 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:57.870929 3382 sys_catalog.cc:565] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:54:57.871308 3355 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:54:57.875062 3355 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 1 FOLLOWER]: Refusing update from remote peer 26c6b20fa203404a88463ea27bdd3c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20261118 02:54:57.875769 3357 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b" is_pre_election: true
I20261118 02:54:57.876155 3499 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:54:57.876289 3357 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 1.
I20261118 02:54:57.876751 3356 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b"
I20261118 02:54:57.889021 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.229.65:0
--local_ip_for_outbound_sockets=127.2.229.65
--webserver_interface=127.2.229.65
--webserver_port=0
--tserver_master_addrs=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20261118 02:54:57.894387 3356 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 1.
I20261118 02:54:57.952503 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 1 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:57.952473 3522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:57.952821 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:57.952841 3522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:58.047472 3529 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:54:58.047583 3529 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: loading cluster ID for follower catalog manager: success
I20261118 02:54:58.051774 3529 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: acquiring CA information for follower catalog manager: success
I20261118 02:54:58.054214 3529 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:54:58.360467 3531 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:54:58.360755 3531 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:58.362532 3531 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
I20261118 02:54:58.362941 3287 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:54:58.363163 3357 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b" is_pre_election: true
I20261118 02:54:58.363919 3403 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:54:58.364320 3531 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:54:58.395880 3522 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:54:58.396143 3522 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:58.397858 3522 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:54:58.398509 3287 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:54:58.398557 3427 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4" is_pre_election: true
I20261118 02:54:58.398777 3427 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 457eb38cba564ecd9f6f11245172260b in term 1.
I20261118 02:54:58.399284 3333 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d
I20261118 02:54:58.399699 3522 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261118 02:54:58.399830 3522 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:54:58.399930 3522 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 1 FOLLOWER]: Advancing to term 2
I20261118 02:54:58.403543 3522 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:58.404337 3522 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 2 election: Requested vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:54:58.404817 3287 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d"
I20261118 02:54:58.404838 3427 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4"
I20261118 02:54:58.405036 3427 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 1 FOLLOWER]: Advancing to term 2
I20261118 02:54:58.407102 3427 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 457eb38cba564ecd9f6f11245172260b in term 2.
I20261118 02:54:58.407513 3333 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d
I20261118 02:54:58.407819 3522 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 2 FOLLOWER]: Leader election won for term 2
I20261118 02:54:58.407998 3522 raft_consensus.cc:705] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 2 LEADER]: Becoming Leader. State: Replica: 457eb38cba564ecd9f6f11245172260b, State: Running, Role: LEADER
I20261118 02:54:58.408460 3522 consensus_queue.cc:238] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:58.410373 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 2 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:58.410581 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
I20261118 02:54:58.412915 3536 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 02:54:58.413828 3427 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 2 FOLLOWER]: Refusing update from remote peer 457eb38cba564ecd9f6f11245172260b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20261118 02:54:58.414433 3514 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:54:58.414793 3514 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:54:58.414903 3514 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20261118 02:54:58.414961 3508 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Connected to new peer: Peer: permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20261118 02:54:58.416915 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 2 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:58.417241 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:58.418393 3287 raft_consensus.cc:3091] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 1 LEADER]: Stepping down as leader of term 1
I20261118 02:54:58.418519 3287 raft_consensus.cc:748] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26c6b20fa203404a88463ea27bdd3c7d, State: Running, Role: LEADER
I20261118 02:54:58.418670 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:58.418843 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
I20261118 02:54:58.418823 3287 consensus_queue.cc:261] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:58.419467 3287 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 1 FOLLOWER]: Advancing to term 2
I20261118 02:54:58.422813 3287 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 2 FOLLOWER]: Refusing update from remote peer 457eb38cba564ecd9f6f11245172260b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20261118 02:54:58.423769 3508 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20261118 02:54:58.426613 3536 catalog_manager.cc:1535] Initializing Kudu cluster ID...
W20261118 02:54:58.428505 3514 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:54:58.428720 3514 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.229.65
I20261118 02:54:58.430495 3536 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:54:58.431577 3536 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:54:58.433426 3536 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:54:58.433965 3499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 2 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:58.434190 3499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:58.434727 3539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:58.435176 3539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:58.436134 3536 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Loaded TSK: 0
I20261118 02:54:58.437395 3536 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 02:54:58.438673 3536 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:54:58.439821 3536 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 02:54:58.440634 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:58.440830 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:58.444355 3514 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.229.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.229.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.229.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:54:58.446568 3514 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:54:58.449016 3514 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:54:58.460449 3541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:58.462440 3542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:54:58.467446 3514 server_base.cc:1061] running on GCE node
W20261118 02:54:58.467799 3544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:54:58.468505 3514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:54:58.469703 3514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:54:58.470932 3514 hybrid_clock.cc:648] HybridClock initialized: now 1794970498470863 us; error 57 us; skew 500 ppm
I20261118 02:54:58.471282 3514 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:54:58.474207 3514 webserver.cc:577] Webserver started at http://127.2.229.65:33489/ using document root <none> and password file <none>
I20261118 02:54:58.474917 3514 fs_manager.cc:362] Metadata directory not provided
I20261118 02:54:58.475003 3514 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:54:58.475374 3514 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 02:54:58.477813 3514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/data/instance:
uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
format_stamp: "Formatted at 2026-11-18 02:54:58 on dist-test-slave-49p1"
I20261118 02:54:58.478971 3514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/instance:
uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
format_stamp: "Formatted at 2026-11-18 02:54:58 on dist-test-slave-49p1"
I20261118 02:54:58.484637 3514 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20261118 02:54:58.488225 3550 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:58.489722 3514 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20261118 02:54:58.489877 3514 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal
uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
format_stamp: "Formatted at 2026-11-18 02:54:58 on dist-test-slave-49p1"
I20261118 02:54:58.490063 3514 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:54:58.502089 3514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:54:58.502875 3514 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:54:58.503103 3514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:54:58.504015 3514 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261118 02:54:58.505708 3514 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261118 02:54:58.505801 3514 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:58.505897 3514 ts_tablet_manager.cc:662] Registered 0 tablets
I20261118 02:54:58.505963 3514 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:58.534147 3487 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:54:58.539775 3493 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:54:58.539885 3493 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: loading cluster ID for follower catalog manager: success
I20261118 02:54:58.543128 3514 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.65:33705
I20261118 02:54:58.543162 3662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.65:33705 every 8 connection(s)
I20261118 02:54:58.543691 3493 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: acquiring CA information for follower catalog manager: success
I20261118 02:54:58.544982 3514 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/data/info.pb
I20261118 02:54:58.546111 3493 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:54:58.547616 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 3514
I20261118 02:54:58.547833 2965 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/instance
I20261118 02:54:58.554724 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.229.66:0
--local_ip_for_outbound_sockets=127.2.229.66
--webserver_interface=127.2.229.66
--webserver_port=0
--tserver_master_addrs=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20261118 02:54:58.567888 3665 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:54:58.568353 3665 heartbeater.cc:462] Registering TS with master...
I20261118 02:54:58.569535 3665 heartbeater.cc:508] Master 127.2.229.125:38339 requested a full tablet report, sending...
I20261118 02:54:58.572042 3346 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:54:58.573827 3663 heartbeater.cc:345] Connected to a master server at 127.2.229.126:38025
I20261118 02:54:58.573948 3663 heartbeater.cc:462] Registering TS with master...
I20261118 02:54:58.574318 3663 heartbeater.cc:508] Master 127.2.229.126:38025 requested a full tablet report, sending...
I20261118 02:54:58.574910 3346 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.2.229.65:52789
I20261118 02:54:58.575831 3664 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:54:58.575831 3277 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:54:58.575954 3664 heartbeater.cc:462] Registering TS with master...
I20261118 02:54:58.576434 3664 heartbeater.cc:508] Master 127.2.229.124:36695 requested a full tablet report, sending...
I20261118 02:54:58.578016 3417 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:54:58.895001 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:58.895264 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
W20261118 02:54:58.903096 3673 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:54:58.903806 3673 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:54:58.903934 3673 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:54:58.912252 3673 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:54:58.912453 3673 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.229.66
I20261118 02:54:58.922938 3673 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.229.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.229.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.229.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:54:58.924654 3673 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:54:58.926412 3673 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:54:58.936448 3678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:58.936859 3679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:58.937366 3681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:54:58.938285 3673 server_base.cc:1061] running on GCE node
I20261118 02:54:58.938798 3673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:54:58.939705 3673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:54:58.940901 3673 hybrid_clock.cc:648] HybridClock initialized: now 1794970498940848 us; error 49 us; skew 500 ppm
I20261118 02:54:58.941256 3673 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:54:58.943318 3673 webserver.cc:577] Webserver started at http://127.2.229.66:44025/ using document root <none> and password file <none>
I20261118 02:54:58.943980 3673 fs_manager.cc:362] Metadata directory not provided
I20261118 02:54:58.944077 3673 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:54:58.944475 3673 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 02:54:58.946802 3673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/data/instance:
uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
format_stamp: "Formatted at 2026-11-18 02:54:58 on dist-test-slave-49p1"
I20261118 02:54:58.947548 3673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/instance:
uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
format_stamp: "Formatted at 2026-11-18 02:54:58 on dist-test-slave-49p1"
I20261118 02:54:58.952814 3673 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20261118 02:54:58.956051 3687 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:58.957516 3673 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20261118 02:54:58.957676 3673 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal
uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
format_stamp: "Formatted at 2026-11-18 02:54:58 on dist-test-slave-49p1"
I20261118 02:54:58.957875 3673 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:54:58.984292 3673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:54:58.985123 3673 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:54:58.985370 3673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:54:58.986326 3673 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261118 02:54:58.988174 3673 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261118 02:54:58.988272 3673 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20261118 02:54:58.988376 3673 ts_tablet_manager.cc:662] Registered 0 tablets
I20261118 02:54:58.988443 3673 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:59.041635 3673 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.66:40377
I20261118 02:54:59.041688 3799 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.66:40377 every 8 connection(s)
I20261118 02:54:59.043529 3673 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/data/info.pb
I20261118 02:54:59.044204 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 3673
I20261118 02:54:59.044379 2965 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/instance
I20261118 02:54:59.049989 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.229.67:0
--local_ip_for_outbound_sockets=127.2.229.67
--webserver_interface=127.2.229.67
--webserver_port=0
--tserver_master_addrs=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20261118 02:54:59.064184 3802 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:54:59.064344 3801 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:54:59.064606 3801 heartbeater.cc:462] Registering TS with master...
I20261118 02:54:59.064667 3800 heartbeater.cc:345] Connected to a master server at 127.2.229.126:38025
I20261118 02:54:59.064613 3802 heartbeater.cc:462] Registering TS with master...
I20261118 02:54:59.064791 3800 heartbeater.cc:462] Registering TS with master...
I20261118 02:54:59.065838 3801 heartbeater.cc:508] Master 127.2.229.124:36695 requested a full tablet report, sending...
I20261118 02:54:59.065869 3802 heartbeater.cc:508] Master 127.2.229.125:38339 requested a full tablet report, sending...
I20261118 02:54:59.066675 3800 heartbeater.cc:508] Master 127.2.229.126:38025 requested a full tablet report, sending...
I20261118 02:54:59.067582 3346 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:54:59.067612 3417 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:54:59.067667 3277 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:54:59.068550 3346 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.2.229.66:38401
I20261118 02:54:59.078799 3665 heartbeater.cc:500] Master 127.2.229.125:38339 was elected leader, sending a full tablet report...
I20261118 02:54:59.384109 3808 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:54:59.384342 3808 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:59.385217 3808 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
I20261118 02:54:59.385731 3287 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:54:59.385867 3357 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b" is_pre_election: true
I20261118 02:54:59.386508 3402 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:54:59.386825 3808 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20261118 02:54:59.393323 3807 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:54:59.393580 3807 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:54:59.393699 3807 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20261118 02:54:59.396874 3812 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:54:59.397053 3812 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:59.397723 3812 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:54:59.398237 3357 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b" is_pre_election: true
I20261118 02:54:59.398368 3427 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4" is_pre_election: true
I20261118 02:54:59.398609 3427 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 2.
I20261118 02:54:59.399027 3263 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4; no voters: 457eb38cba564ecd9f6f11245172260b
I20261118 02:54:59.399289 3812 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261118 02:54:59.399396 3812 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:54:59.399488 3812 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 2 FOLLOWER]: Advancing to term 3
I20261118 02:54:59.401449 3812 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:59.401973 3812 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 3 election: Requested vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:54:59.402426 3357 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b"
I20261118 02:54:59.402446 3427 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4"
I20261118 02:54:59.402653 3427 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 2 FOLLOWER]: Advancing to term 3
W20261118 02:54:59.404016 3807 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:54:59.404276 3807 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.229.67
I20261118 02:54:59.405571 3427 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 3.
I20261118 02:54:59.405942 3263 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4; no voters: 457eb38cba564ecd9f6f11245172260b
I20261118 02:54:59.406261 3812 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 3 FOLLOWER]: Leader election won for term 3
I20261118 02:54:59.406652 3812 raft_consensus.cc:705] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 3 LEADER]: Becoming Leader. State: Replica: 26c6b20fa203404a88463ea27bdd3c7d, State: Running, Role: LEADER
I20261118 02:54:59.406944 3812 consensus_queue.cc:238] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:59.407971 3813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:59.408356 3813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
I20261118 02:54:59.409422 3427 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 3 FOLLOWER]: Refusing update from remote peer 26c6b20fa203404a88463ea27bdd3c7d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20261118 02:54:59.415213 3814 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261118 02:54:59.415520 3357 raft_consensus.cc:3091] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 2 LEADER]: Stepping down as leader of term 2
I20261118 02:54:59.415633 3357 raft_consensus.cc:748] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 457eb38cba564ecd9f6f11245172260b, State: Running, Role: LEADER
I20261118 02:54:59.415952 3357 consensus_queue.cc:261] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:54:59.416488 3357 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 2 FOLLOWER]: Advancing to term 3
I20261118 02:54:59.419521 3357 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 3 FOLLOWER]: Refusing update from remote peer 26c6b20fa203404a88463ea27bdd3c7d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20261118 02:54:59.420855 3807 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.229.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.229.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.229.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:54:59.422152 3808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:59.422389 3808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:59.422694 3807 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:54:59.424437 3814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:59.424744 3814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
I20261118 02:54:59.424973 3817 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261118 02:54:59.425695 3807 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261118 02:54:59.436048 3816 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 02:54:59.436496 3808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:59.436740 3808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:59.438357 3824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:59.438558 3816 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 02:54:59.438495 3820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:59.438706 3820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:59.438733 3824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:54:59.439785 3816 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:54:59.439894 3816 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:54:59.441496 3816 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:54:59.442557 3816 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Loaded TSK: 0
I20261118 02:54:59.443387 3816 catalog_manager.cc:1565] Initializing in-progress tserver states...
W20261118 02:54:59.443599 3825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:54:59.443758 3826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:54:59.444406 3816 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:54:59.445526 3816 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 02:54:59.448345 3807 server_base.cc:1061] running on GCE node
W20261118 02:54:59.449287 3828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:54:59.449965 3807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:54:59.451016 3807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:54:59.452230 3807 hybrid_clock.cc:648] HybridClock initialized: now 1794970499452164 us; error 51 us; skew 500 ppm
I20261118 02:54:59.452565 3807 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:54:59.455207 3807 webserver.cc:577] Webserver started at http://127.2.229.67:39695/ using document root <none> and password file <none>
I20261118 02:54:59.455945 3807 fs_manager.cc:362] Metadata directory not provided
I20261118 02:54:59.456046 3807 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:54:59.456454 3807 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 02:54:59.458799 3807 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/data/instance:
uuid: "3f1973986ec1466eab5bec01dd81654f"
format_stamp: "Formatted at 2026-11-18 02:54:59 on dist-test-slave-49p1"
I20261118 02:54:59.459549 3807 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/instance:
uuid: "3f1973986ec1466eab5bec01dd81654f"
format_stamp: "Formatted at 2026-11-18 02:54:59 on dist-test-slave-49p1"
I20261118 02:54:59.465080 3807 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20261118 02:54:59.468488 3834 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:59.469940 3807 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20261118 02:54:59.470105 3807 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal
uuid: "3f1973986ec1466eab5bec01dd81654f"
format_stamp: "Formatted at 2026-11-18 02:54:59 on dist-test-slave-49p1"
I20261118 02:54:59.470289 3807 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:54:59.494714 3807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:54:59.495630 3807 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:54:59.495878 3807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:54:59.496881 3807 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261118 02:54:59.498534 3807 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261118 02:54:59.498627 3807 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:59.498729 3807 ts_tablet_manager.cc:662] Registered 0 tablets
I20261118 02:54:59.498799 3807 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261118 02:54:59.535753 3807 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.67:42237
I20261118 02:54:59.535802 3946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.67:42237 every 8 connection(s)
I20261118 02:54:59.537293 3807 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/data/info.pb
I20261118 02:54:59.540560 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 3807
I20261118 02:54:59.540755 2965 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/instance
I20261118 02:54:59.555756 3949 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:54:59.556232 3949 heartbeater.cc:462] Registering TS with master...
I20261118 02:54:59.557366 3949 heartbeater.cc:508] Master 127.2.229.125:38339 requested a full tablet report, sending...
I20261118 02:54:59.559212 3346 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:54:59.561666 3948 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:54:59.561831 3948 heartbeater.cc:462] Registering TS with master...
I20261118 02:54:59.562374 3948 heartbeater.cc:508] Master 127.2.229.124:36695 requested a full tablet report, sending...
I20261118 02:54:59.563476 3417 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:54:59.563777 3947 heartbeater.cc:345] Connected to a master server at 127.2.229.126:38025
I20261118 02:54:59.563903 3947 heartbeater.cc:462] Registering TS with master...
I20261118 02:54:59.564309 3947 heartbeater.cc:508] Master 127.2.229.126:38025 requested a full tablet report, sending...
I20261118 02:54:59.565203 3277 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:54:59.567343 3277 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.2.229.67:38945
I20261118 02:54:59.570621 3802 heartbeater.cc:500] Master 127.2.229.125:38339 was elected leader, sending a full tablet report...
I20261118 02:54:59.575372 2965 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20261118 02:54:59.609403 3277 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_dc3979f8c8b148caa37b0a8ef420ba9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:54:59.609889 3276 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_df2a47bb353f4cdcb4c16399a99cb722"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:54:59.610028 3274 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_0006608f28004098bb49ed40308f98fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:54:59.610365 3273 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_f777ce1b15a4431fa208932de26f41fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:54:59.610468 3272 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_295a3e7e0d1f4e9db7078f6e5a5b9da1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:54:59.611837 3270 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_bca0059ea2004174b89866f3e8cc68fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:54:59.612268 3268 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_287099a68a3540f1be0284c71d4d7056"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:54:59.613899 3271 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_a4f08da750c54c49b9b7d73ebcf52624"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:54:59.615154 3275 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_c113c2b57e17479e92e241234a241af7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:54:59.614686 3269 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_28c62f411da74545aa42dad7325bc064"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261118 02:54:59.619489 3277 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dc3979f8c8b148caa37b0a8ef420ba9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261118 02:54:59.636780 2965 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task8fD3U4/build/asan/bin/kudu pid 3383
I20261118 02:54:59.638343 3447 generic_service.cc:196] Checking for leaks (request via RPC)
I20261118 02:54:59.815558 2965 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task8fD3U4/build/asan/bin/kudu with pid 3383
I20261118 02:54:59.842792 3817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:54:59.843138 3817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
W20261118 02:54:59.940655 3263 consensus_peers.cc:597] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d -> Peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Couldn't send request to peer 813a6937cea442ef81645fccc32207f4. Status: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:00.052235 3277 catalog_manager.cc:2215] created HMS catalog entry for table [id=052f3134115244a6a2443e96121313e6]
W20261118 02:55:00.057281 3274 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0006608f28004098bb49ed40308f98fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:00.058568 3276 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_df2a47bb353f4cdcb4c16399a99cb722 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:00.059717 3273 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f777ce1b15a4431fa208932de26f41fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:00.068867 3948 heartbeater.cc:648] Failed to heartbeat to 127.2.229.124:36695 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:00.070811 3947 heartbeater.cc:500] Master 127.2.229.126:38025 was elected leader, sending a full tablet report...
I20261118 02:55:00.072726 3800 heartbeater.cc:500] Master 127.2.229.126:38025 was elected leader, sending a full tablet report...
W20261118 02:55:00.075165 3801 heartbeater.cc:648] Failed to heartbeat to 127.2.229.124:36695 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:00.081283 3663 heartbeater.cc:500] Master 127.2.229.126:38025 was elected leader, sending a full tablet report...
W20261118 02:55:00.087898 3664 heartbeater.cc:648] Failed to heartbeat to 127.2.229.124:36695 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:00.123944 3277 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_dc3979f8c8b148caa37b0a8ef420ba9c" } new_table_name: "default.Table_8ae376d4d4d64414b5c08dc3c2e58bc4" modify_external_catalogs: true
W20261118 02:55:00.141810 3272 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_295a3e7e0d1f4e9db7078f6e5a5b9da1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:00.142956 3270 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bca0059ea2004174b89866f3e8cc68fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:00.143929 3271 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a4f08da750c54c49b9b7d73ebcf52624 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:00.144840 3275 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c113c2b57e17479e92e241234a241af7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:00.145725 3268 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_287099a68a3540f1be0284c71d4d7056 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:00.146646 3269 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_28c62f411da74545aa42dad7325bc064 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261118 02:55:00.149622 3274 catalog_manager.cc:2215] created HMS catalog entry for table [id=a2ece7af36934cdf80ad8458024adec7]
I20261118 02:55:00.217029 3274 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_0006608f28004098bb49ed40308f98fe" } new_table_name: "default.Table_4f7796d5d5d54def843b971e66d15ea6" modify_external_catalogs: true
I20261118 02:55:00.230227 3276 catalog_manager.cc:2215] created HMS catalog entry for table [id=8a5ea4b95bee4db7af9ba875250d564b]
I20261118 02:55:00.244421 3276 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_df2a47bb353f4cdcb4c16399a99cb722" } new_table_name: "default.Table_0385cef6593c419eb426657c8cbef2bf" modify_external_catalogs: true
I20261118 02:55:00.245314 3881 tablet_service.cc:1500] Processing CreateTablet for tablet 9da387134c724364a6c46f65f4e3d605 (DEFAULT_TABLE table=default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.246308 3882 tablet_service.cc:1500] Processing CreateTablet for tablet d35ba6ef82f9493ca4605294f109b5dc (DEFAULT_TABLE table=default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.247421 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35ba6ef82f9493ca4605294f109b5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.247543 3598 tablet_service.cc:1500] Processing CreateTablet for tablet d35ba6ef82f9493ca4605294f109b5dc (DEFAULT_TABLE table=default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.249102 3597 tablet_service.cc:1500] Processing CreateTablet for tablet 9da387134c724364a6c46f65f4e3d605 (DEFAULT_TABLE table=default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.249977 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da387134c724364a6c46f65f4e3d605. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.252460 3880 tablet_service.cc:1500] Processing CreateTablet for tablet 1d347c0ae53241708ec623562f4740d4 (DEFAULT_TABLE table=default.table_df2a47bb353f4cdcb4c16399a99cb722 [id=8a5ea4b95bee4db7af9ba875250d564b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.252686 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da387134c724364a6c46f65f4e3d605. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.253087 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d347c0ae53241708ec623562f4740d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.256911 3596 tablet_service.cc:1500] Processing CreateTablet for tablet 1d347c0ae53241708ec623562f4740d4 (DEFAULT_TABLE table=default.table_df2a47bb353f4cdcb4c16399a99cb722 [id=8a5ea4b95bee4db7af9ba875250d564b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.257423 3596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d347c0ae53241708ec623562f4740d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.266620 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35ba6ef82f9493ca4605294f109b5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.275245 3735 tablet_service.cc:1500] Processing CreateTablet for tablet d35ba6ef82f9493ca4605294f109b5dc (DEFAULT_TABLE table=default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.276314 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 9da387134c724364a6c46f65f4e3d605 (DEFAULT_TABLE table=default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.276580 3733 tablet_service.cc:1500] Processing CreateTablet for tablet 1d347c0ae53241708ec623562f4740d4 (DEFAULT_TABLE table=default.table_df2a47bb353f4cdcb4c16399a99cb722 [id=8a5ea4b95bee4db7af9ba875250d564b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.277709 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d347c0ae53241708ec623562f4740d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.278225 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da387134c724364a6c46f65f4e3d605. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.280815 4012 tablet_bootstrap.cc:492] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.281162 3735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35ba6ef82f9493ca4605294f109b5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.288311 4012 tablet_bootstrap.cc:654] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.290895 4011 tablet_bootstrap.cc:492] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.294698 4011 tablet_bootstrap.cc:654] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.295320 4012 log.cc:826] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:00.297165 4013 tablet_bootstrap.cc:492] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.297922 4011 log.cc:826] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:00.300823 4013 tablet_bootstrap.cc:654] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.302726 4013 log.cc:826] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:00.314244 4011 tablet_bootstrap.cc:492] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.314612 4011 ts_tablet_manager.cc:1449] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.002s
I20261118 02:55:00.317462 4012 tablet_bootstrap.cc:492] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.320417 4012 ts_tablet_manager.cc:1449] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.005s
I20261118 02:55:00.321329 4013 tablet_bootstrap.cc:492] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.321704 4013 ts_tablet_manager.cc:1449] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.004s
I20261118 02:55:00.327342 4012 raft_consensus.cc:360] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.327785 4012 raft_consensus.cc:386] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.327900 4012 raft_consensus.cc:748] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.329339 4012 consensus_queue.cc:261] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.329422 4013 raft_consensus.cc:360] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.330000 4013 raft_consensus.cc:386] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.330142 4013 raft_consensus.cc:748] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.330996 4013 consensus_queue.cc:261] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.335538 3273 catalog_manager.cc:2215] created HMS catalog entry for table [id=4a09f210503b499f8e842f6daafb33da]
I20261118 02:55:00.349604 3273 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_22f31dd1ff9e4185a6618f11b5493434"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:00.350908 4011 raft_consensus.cc:360] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.351361 4011 raft_consensus.cc:386] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.351476 4011 raft_consensus.cc:748] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.352247 4011 consensus_queue.cc:261] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.358886 3882 tablet_service.cc:1500] Processing CreateTablet for tablet 4c230b673b8f487fbc79bdfb11a48628 (DEFAULT_TABLE table=default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.358867 3596 tablet_service.cc:1500] Processing CreateTablet for tablet 4c230b673b8f487fbc79bdfb11a48628 (DEFAULT_TABLE table=default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.359292 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c230b673b8f487fbc79bdfb11a48628. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.359292 3596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c230b673b8f487fbc79bdfb11a48628. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.359711 4011 ts_tablet_manager.cc:1480] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.045s user 0.009s sys 0.008s
I20261118 02:55:00.360901 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 4c230b673b8f487fbc79bdfb11a48628 (DEFAULT_TABLE table=default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.361301 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c230b673b8f487fbc79bdfb11a48628. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.363598 4011 tablet_bootstrap.cc:492] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.365597 4011 tablet_bootstrap.cc:654] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.369576 4012 ts_tablet_manager.cc:1480] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.049s user 0.014s sys 0.001s
I20261118 02:55:00.372473 4012 tablet_bootstrap.cc:492] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.374375 4012 tablet_bootstrap.cc:654] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.376399 4011 tablet_bootstrap.cc:492] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.376606 4011 ts_tablet_manager.cc:1449] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20261118 02:55:00.377655 4013 ts_tablet_manager.cc:1480] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.056s user 0.010s sys 0.013s
I20261118 02:55:00.378535 3813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:00.378729 3813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:00.378830 4013 tablet_bootstrap.cc:492] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.383507 4013 tablet_bootstrap.cc:654] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.383839 4011 raft_consensus.cc:360] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.384505 4011 raft_consensus.cc:386] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.384788 4011 raft_consensus.cc:748] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.386204 4011 consensus_queue.cc:261] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.388906 4011 ts_tablet_manager.cc:1480] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20261118 02:55:00.390585 4011 tablet_bootstrap.cc:492] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.393182 4011 tablet_bootstrap.cc:654] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.393857 4012 tablet_bootstrap.cc:492] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.394022 4012 ts_tablet_manager.cc:1449] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20261118 02:55:00.394488 4013 tablet_bootstrap.cc:492] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.394627 4013 ts_tablet_manager.cc:1449] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20261118 02:55:00.395565 4013 raft_consensus.cc:360] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.395723 4013 raft_consensus.cc:386] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.395813 4013 raft_consensus.cc:748] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.396037 4013 consensus_queue.cc:261] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.396378 4012 raft_consensus.cc:360] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.396605 4013 ts_tablet_manager.cc:1480] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:00.396615 4012 raft_consensus.cc:386] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.396886 4013 tablet_bootstrap.cc:492] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.397018 4012 raft_consensus.cc:748] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.397346 4012 consensus_queue.cc:261] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.397943 4012 ts_tablet_manager.cc:1480] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 02:55:00.398690 4013 tablet_bootstrap.cc:654] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.400358 4012 tablet_bootstrap.cc:492] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.401256 4011 tablet_bootstrap.cc:492] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.401438 4011 ts_tablet_manager.cc:1449] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20261118 02:55:00.402194 4011 raft_consensus.cc:360] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.402381 4011 raft_consensus.cc:386] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.402462 4011 raft_consensus.cc:748] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.402734 4011 consensus_queue.cc:261] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.403929 4012 tablet_bootstrap.cc:654] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.405066 4011 ts_tablet_manager.cc:1480] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20261118 02:55:00.407989 4011 tablet_bootstrap.cc:492] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.409744 4011 tablet_bootstrap.cc:654] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.412372 4011 tablet_bootstrap.cc:492] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.412524 4011 ts_tablet_manager.cc:1449] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20261118 02:55:00.413409 4011 raft_consensus.cc:360] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.413542 4011 raft_consensus.cc:386] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.413595 4011 raft_consensus.cc:748] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.413780 4011 consensus_queue.cc:261] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.414228 4011 ts_tablet_manager.cc:1480] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 02:55:00.415045 4013 tablet_bootstrap.cc:492] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.415215 4013 ts_tablet_manager.cc:1449] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.012s
I20261118 02:55:00.417480 4013 raft_consensus.cc:360] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.417682 4013 raft_consensus.cc:386] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.417820 4013 raft_consensus.cc:748] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.418079 4013 consensus_queue.cc:261] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.418542 4013 ts_tablet_manager.cc:1480] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20261118 02:55:00.418869 4013 tablet_bootstrap.cc:492] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.420616 4013 tablet_bootstrap.cc:654] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.424837 4012 tablet_bootstrap.cc:492] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.424872 4013 tablet_bootstrap.cc:492] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.424998 4013 ts_tablet_manager.cc:1449] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20261118 02:55:00.425041 4012 ts_tablet_manager.cc:1449] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.013s
I20261118 02:55:00.425782 4012 raft_consensus.cc:360] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.425794 4013 raft_consensus.cc:360] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.425967 4013 raft_consensus.cc:386] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.425995 4012 raft_consensus.cc:386] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.426057 4013 raft_consensus.cc:748] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.426084 4012 raft_consensus.cc:748] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.426292 4013 consensus_queue.cc:261] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.426332 4012 consensus_queue.cc:261] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.426821 4012 ts_tablet_manager.cc:1480] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 02:55:00.427209 4012 tablet_bootstrap.cc:492] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.429081 4012 tablet_bootstrap.cc:654] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.431933 4013 ts_tablet_manager.cc:1480] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20261118 02:55:00.437523 4017 raft_consensus.cc:499] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:00.437824 4017 raft_consensus.cc:521] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.438512 4012 tablet_bootstrap.cc:492] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.438681 4012 ts_tablet_manager.cc:1449] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20261118 02:55:00.439566 4012 raft_consensus.cc:360] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.439760 4012 raft_consensus.cc:386] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.439862 4012 raft_consensus.cc:748] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.440130 4012 consensus_queue.cc:261] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.441303 4017 leader_election.cc:290] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:00.450096 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1d347c0ae53241708ec623562f4740d4" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:00.450697 3755 raft_consensus.cc:2490] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:00.451416 3552 leader_election.cc:304] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:00.451931 4017 raft_consensus.cc:2826] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:00.452076 4017 raft_consensus.cc:499] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:00.452214 4017 raft_consensus.cc:3096] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.455505 4017 raft_consensus.cc:521] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.455982 3272 catalog_manager.cc:2215] created HMS catalog entry for table [id=44b390866720455b8d2714eae004383b]
I20261118 02:55:00.456753 4017 leader_election.cc:290] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:00.457056 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1d347c0ae53241708ec623562f4740d4" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:00.457269 3755 raft_consensus.cc:3096] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.460831 3755 raft_consensus.cc:2490] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:00.461428 3552 leader_election.cc:304] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:00.461755 4017 raft_consensus.cc:2826] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:00.461974 4017 raft_consensus.cc:705] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:00.462618 4017 consensus_queue.cc:238] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.467639 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1d347c0ae53241708ec623562f4740d4" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:00.468195 3902 raft_consensus.cc:2490] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:00.469256 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1d347c0ae53241708ec623562f4740d4" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:00.469559 3901 raft_consensus.cc:3096] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.479483 4012 ts_tablet_manager.cc:1480] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.041s user 0.019s sys 0.019s
I20261118 02:55:00.482509 3901 raft_consensus.cc:2490] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:00.487835 3734 tablet_service.cc:1500] Processing CreateTablet for tablet a95a3f25901e468ab6e6cd02c2cd035f (DEFAULT_TABLE table=default.table_295a3e7e0d1f4e9db7078f6e5a5b9da1 [id=44b390866720455b8d2714eae004383b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.488382 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95a3f25901e468ab6e6cd02c2cd035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.492393 4017 consensus_queue.cc:1049] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:00.493752 3272 master_service.cc:965] ReplaceTablet: received request to replace tablet 9da387134c724364a6c46f65f4e3d605 from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:00.494998 3882 tablet_service.cc:1500] Processing CreateTablet for tablet a95a3f25901e468ab6e6cd02c2cd035f (DEFAULT_TABLE table=default.table_295a3e7e0d1f4e9db7078f6e5a5b9da1 [id=44b390866720455b8d2714eae004383b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.495446 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95a3f25901e468ab6e6cd02c2cd035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.496140 3596 tablet_service.cc:1500] Processing CreateTablet for tablet a95a3f25901e468ab6e6cd02c2cd035f (DEFAULT_TABLE table=default.table_295a3e7e0d1f4e9db7078f6e5a5b9da1 [id=44b390866720455b8d2714eae004383b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.496567 3596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95a3f25901e468ab6e6cd02c2cd035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.497073 4017 consensus_queue.cc:1049] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:00.502254 4013 tablet_bootstrap.cc:492] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.503224 4012 tablet_bootstrap.cc:492] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.504213 4013 tablet_bootstrap.cc:654] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.505105 4012 tablet_bootstrap.cc:654] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.510735 3272 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet 9da387134c724364a6c46f65f4e3d605
I20261118 02:55:00.511519 3272 catalog_manager.cc:6936] ReplaceTablet: tablet 9da387134c724364a6c46f65f4e3d605 deleted and replaced by tablet 6bf520b76802491da8e3b090ad2152da
I20261118 02:55:00.511839 3596 tablet_service.cc:1547] Processing DeleteTablet for tablet 9da387134c724364a6c46f65f4e3d605 with delete_type TABLET_DATA_DELETED (replaced by tablet 6bf520b76802491da8e3b090ad2152da) from {username='slave'} at 127.0.0.1:52990
I20261118 02:55:00.512848 4040 tablet_replica.cc:333] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:00.513280 4040 raft_consensus.cc:2261] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:00.513070 3272 catalog_manager.cc:5840] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: UNKNOWN } } }
I20261118 02:55:00.515702 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet 9da387134c724364a6c46f65f4e3d605 with delete_type TABLET_DATA_DELETED (replaced by tablet 6bf520b76802491da8e3b090ad2152da) from {username='slave'} at 127.0.0.1:50234
I20261118 02:55:00.515880 3733 tablet_service.cc:1547] Processing DeleteTablet for tablet 9da387134c724364a6c46f65f4e3d605 with delete_type TABLET_DATA_DELETED (replaced by tablet 6bf520b76802491da8e3b090ad2152da) from {username='slave'} at 127.0.0.1:57880
I20261118 02:55:00.517897 4011 tablet_bootstrap.cc:492] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.519800 4011 tablet_bootstrap.cc:654] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.532027 3596 tablet_service.cc:1500] Processing CreateTablet for tablet 6bf520b76802491da8e3b090ad2152da (DEFAULT_TABLE table=default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.532541 3596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bf520b76802491da8e3b090ad2152da. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.533190 4011 tablet_bootstrap.cc:492] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.533378 4011 ts_tablet_manager.cc:1449] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20261118 02:55:00.534143 4011 raft_consensus.cc:360] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.534345 4011 raft_consensus.cc:386] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.534431 4011 raft_consensus.cc:748] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.534718 4040 raft_consensus.cc:2290] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:00.534682 4011 consensus_queue.cc:261] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.535347 4011 ts_tablet_manager.cc:1480] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20261118 02:55:00.537659 4042 tablet_replica.cc:333] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:00.538035 4043 tablet_replica.cc:333] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:00.538187 4042 raft_consensus.cc:2261] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:00.538550 4043 raft_consensus.cc:2261] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:00.538619 4042 raft_consensus.cc:2290] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:00.538967 4043 raft_consensus.cc:2290] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:00.540051 4040 ts_tablet_manager.cc:1962] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:00.541092 4012 tablet_bootstrap.cc:492] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.541311 4012 ts_tablet_manager.cc:1449] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.010s
W20261118 02:55:00.540409 3950 tablet.cc:2441] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261118 02:55:00.543677 4043 ts_tablet_manager.cc:1962] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:00.542219 4012 raft_consensus.cc:360] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.544047 4012 raft_consensus.cc:386] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.544211 4012 raft_consensus.cc:748] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.545596 3733 tablet_service.cc:1500] Processing CreateTablet for tablet 6bf520b76802491da8e3b090ad2152da (DEFAULT_TABLE table=default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.546067 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bf520b76802491da8e3b090ad2152da. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.546165 4042 ts_tablet_manager.cc:1962] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:00.547101 4012 consensus_queue.cc:261] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.553028 4043 ts_tablet_manager.cc:1975] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 02:55:00.553257 4043 log.cc:1199] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/9da387134c724364a6c46f65f4e3d605
I20261118 02:55:00.554076 4043 ts_tablet_manager.cc:1996] T 9da387134c724364a6c46f65f4e3d605 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:00.554884 3272 master_service.cc:965] ReplaceTablet: received request to replace tablet d35ba6ef82f9493ca4605294f109b5dc from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:00.556187 3882 tablet_service.cc:1500] Processing CreateTablet for tablet 6bf520b76802491da8e3b090ad2152da (DEFAULT_TABLE table=default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.556603 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bf520b76802491da8e3b090ad2152da. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.556599 3262 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet 9da387134c724364a6c46f65f4e3d605 (table default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]) successfully deleted
W20261118 02:55:00.557857 3803 tablet.cc:2441] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261118 02:55:00.561553 3272 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet d35ba6ef82f9493ca4605294f109b5dc
I20261118 02:55:00.562235 3733 tablet_service.cc:1547] Processing DeleteTablet for tablet d35ba6ef82f9493ca4605294f109b5dc with delete_type TABLET_DATA_DELETED (replaced by tablet 4ea7c6ad69d94b06bb0e72468db5c834) from {username='slave'} at 127.0.0.1:57880
I20261118 02:55:00.562553 4043 tablet_replica.cc:333] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:00.562843 4043 raft_consensus.cc:2261] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:00.562845 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet d35ba6ef82f9493ca4605294f109b5dc with delete_type TABLET_DATA_DELETED (replaced by tablet 4ea7c6ad69d94b06bb0e72468db5c834) from {username='slave'} at 127.0.0.1:50234
I20261118 02:55:00.563090 4043 raft_consensus.cc:2290] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:00.563697 4018 raft_consensus.cc:499] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:00.563844 4043 ts_tablet_manager.cc:1962] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:00.563867 4018 raft_consensus.cc:521] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.565840 4018 leader_election.cc:290] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:00.566346 3272 catalog_manager.cc:6936] ReplaceTablet: tablet d35ba6ef82f9493ca4605294f109b5dc deleted and replaced by tablet 4ea7c6ad69d94b06bb0e72468db5c834
I20261118 02:55:00.568436 3272 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_f777ce1b15a4431fa208932de26f41fb" } new_table_name: "default.Table_a7f3710d39fb447da94e16d4ab3933fd" modify_external_catalogs: true
I20261118 02:55:00.570577 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.124:36695
--webserver_interface=127.2.229.124
--webserver_port=39557
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 02:55:00.570830 4043 ts_tablet_manager.cc:1975] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 02:55:00.570952 4043 log.cc:1199] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/d35ba6ef82f9493ca4605294f109b5dc
I20261118 02:55:00.571225 4043 ts_tablet_manager.cc:1996] T d35ba6ef82f9493ca4605294f109b5dc P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:00.573207 3262 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet d35ba6ef82f9493ca4605294f109b5dc (table default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6]) successfully deleted
I20261118 02:55:00.580549 3733 tablet_service.cc:1500] Processing CreateTablet for tablet 4ea7c6ad69d94b06bb0e72468db5c834 (DEFAULT_TABLE table=default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.581049 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea7c6ad69d94b06bb0e72468db5c834. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.583196 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c230b673b8f487fbc79bdfb11a48628" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:00.583457 3902 raft_consensus.cc:2490] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:00.584059 3690 leader_election.cc:304] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:00.584497 4018 raft_consensus.cc:2826] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:00.584605 4018 raft_consensus.cc:499] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:00.584692 4018 raft_consensus.cc:3096] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.585318 3881 tablet_service.cc:1500] Processing CreateTablet for tablet 4ea7c6ad69d94b06bb0e72468db5c834 (DEFAULT_TABLE table=default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.585775 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea7c6ad69d94b06bb0e72468db5c834. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.586014 4013 tablet_bootstrap.cc:492] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.586190 4013 ts_tablet_manager.cc:1449] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.006s
I20261118 02:55:00.586987 4013 raft_consensus.cc:360] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.587194 4013 raft_consensus.cc:386] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.587275 4013 raft_consensus.cc:748] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.587484 4018 raft_consensus.cc:521] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.587584 4013 consensus_queue.cc:261] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.588265 4018 leader_election.cc:290] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:00.589015 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c230b673b8f487fbc79bdfb11a48628" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:00.589186 3902 raft_consensus.cc:3096] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.592068 3902 raft_consensus.cc:2490] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:00.592625 3690 leader_election.cc:304] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:00.595315 4018 raft_consensus.cc:2826] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:00.595502 4018 raft_consensus.cc:705] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:00.596016 4018 consensus_queue.cc:238] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.596525 4011 tablet_bootstrap.cc:492] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.598263 4011 tablet_bootstrap.cc:654] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
W20261118 02:55:00.602990 3666 tablet.cc:2441] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261118 02:55:00.603596 4012 ts_tablet_manager.cc:1480] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20261118 02:55:00.607456 3598 tablet_service.cc:1547] Processing DeleteTablet for tablet d35ba6ef82f9493ca4605294f109b5dc with delete_type TABLET_DATA_DELETED (replaced by tablet 4ea7c6ad69d94b06bb0e72468db5c834) from {username='slave'} at 127.0.0.1:52990
I20261118 02:55:00.607466 3270 catalog_manager.cc:2215] created HMS catalog entry for table [id=469b750094124f159d835e6fb5a6486e]
I20261118 02:55:00.608688 3597 tablet_service.cc:1500] Processing CreateTablet for tablet 4ea7c6ad69d94b06bb0e72468db5c834 (DEFAULT_TABLE table=default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.612509 4013 ts_tablet_manager.cc:1480] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.026s user 0.003s sys 0.003s
I20261118 02:55:00.612942 4013 tablet_bootstrap.cc:492] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.614796 4013 tablet_bootstrap.cc:654] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.614872 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea7c6ad69d94b06bb0e72468db5c834. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.620771 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c230b673b8f487fbc79bdfb11a48628" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:00.621204 3618 raft_consensus.cc:2490] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:00.621644 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c230b673b8f487fbc79bdfb11a48628" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:00.621901 3617 raft_consensus.cc:3096] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.632297 4034 raft_consensus.cc:499] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:00.632472 4034 raft_consensus.cc:521] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.633639 3270 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 8) took 1022 ms. Trace:
I20261118 02:55:00.633811 3617 raft_consensus.cc:2490] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:00.633805 3270 rpcz_store.cc:276] 1118 02:54:59.611330 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:54:59.611623 (+ 293us) service_pool.cc:224] Handling call
1118 02:55:00.143087 (+531464us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:00.143516 (+ 429us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:00.607410 (+463894us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:00.633348 (+ 25938us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:00.633464 (+ 116us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:00.633590 (+ 126us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:00.608348 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:00.608487 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:00.608494 (+ 7us) write_op.cc:435] Acquired schema lock
1118 02:55:00.608497 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:00.608614 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:00.608629 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:00.608635 (+ 6us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:00.608735 (+ 100us) tablet.cc:709] Row locks acquired
1118 02:55:00.608737 (+ 2us) write_op.cc:260] PREPARE: finished
1118 02:55:00.608782 (+ 45us) write_op.cc:270] Start()
1118 02:55:00.608820 (+ 38us) write_op.cc:276] Timestamp: P: 1794970500608749 usec, L: 0
1118 02:55:00.608824 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:00.609035 (+ 211us) log.cc:844] Serialized 432 byte log entry
1118 02:55:00.630464 (+ 21429us) op_driver.cc:466] REPLICATION: finished
1118 02:55:00.632560 (+ 2096us) write_op.cc:301] APPLY: starting
1118 02:55:00.632579 (+ 19us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:00.632621 (+ 42us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:00.632648 (+ 27us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:00.632762 (+ 114us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:00.632795 (+ 33us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:00.632802 (+ 7us) write_op.cc:312] APPLY: finished
1118 02:55:00.632896 (+ 94us) log.cc:844] Serialized 28 byte log entry
1118 02:55:00.632958 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:00.632978 (+ 20us) write_op.cc:454] Released schema lock
1118 02:55:00.632985 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:00.633053 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:00.633057 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":312250,"Hive Metastore.run_cpu_time_us":268,"Hive Metastore.run_wall_time_us":151424,"child_traces":[["op",{"apply.queue_time_us":2030,"apply.run_cpu_time_us":492,"apply.run_wall_time_us":491,"num_ops":2,"prepare.queue_time_us":74,"prepare.run_cpu_time_us":817,"prepare.run_wall_time_us":816,"raft.queue_time_us":168,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":307,"replication_time_us":21599}]]}
I20261118 02:55:00.636461 4034 leader_election.cc:290] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:00.641036 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a95a3f25901e468ab6e6cd02c2cd035f" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:00.641340 3902 raft_consensus.cc:2490] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:00.642102 3553 leader_election.cc:304] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:00.642439 4034 raft_consensus.cc:2826] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:00.642531 4034 raft_consensus.cc:499] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:00.642597 4034 raft_consensus.cc:3096] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.644318 3598 tablet_service.cc:1500] Processing CreateTablet for tablet 9a19cb906c974009aa9df99a14a78024 (DEFAULT_TABLE table=default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.644881 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a19cb906c974009aa9df99a14a78024. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.644904 3882 tablet_service.cc:1500] Processing CreateTablet for tablet 9a19cb906c974009aa9df99a14a78024 (DEFAULT_TABLE table=default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.645284 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a19cb906c974009aa9df99a14a78024. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.645689 4034 raft_consensus.cc:521] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.647038 3270 master_service.cc:965] ReplaceTablet: received request to replace tablet d35ba6ef82f9493ca4605294f109b5dc from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:00.647490 4034 leader_election.cc:290] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:00.647536 3270 master_service.cc:965] ReplaceTablet: received request to replace tablet d35ba6ef82f9493ca4605294f109b5dc from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:00.647471 3733 tablet_service.cc:1500] Processing CreateTablet for tablet 9a19cb906c974009aa9df99a14a78024 (DEFAULT_TABLE table=default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.647881 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a19cb906c974009aa9df99a14a78024. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.647936 3270 master_service.cc:965] ReplaceTablet: received request to replace tablet d35ba6ef82f9493ca4605294f109b5dc from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:00.648491 4012 tablet_bootstrap.cc:492] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.648464 3270 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_a14cb6414555446286bf35bc5c6fb910"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:00.650183 4012 tablet_bootstrap.cc:654] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.658237 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a95a3f25901e468ab6e6cd02c2cd035f" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:00.660755 3902 raft_consensus.cc:3096] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.664189 3902 raft_consensus.cc:2490] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:00.664741 3553 leader_election.cc:304] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:00.665130 4034 raft_consensus.cc:2826] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:00.666065 4034 raft_consensus.cc:705] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:00.666419 4034 consensus_queue.cc:238] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.673485 4034 consensus_queue.cc:1049] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:00.700728 4011 tablet_bootstrap.cc:492] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.701049 4011 ts_tablet_manager.cc:1449] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.105s user 0.021s sys 0.016s
I20261118 02:55:00.701659 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a95a3f25901e468ab6e6cd02c2cd035f" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:00.701901 3755 raft_consensus.cc:2490] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:00.702028 4011 raft_consensus.cc:360] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.702275 4011 raft_consensus.cc:386] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.702410 4011 raft_consensus.cc:748] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.702527 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a95a3f25901e468ab6e6cd02c2cd035f" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:00.702699 3755 raft_consensus.cc:3096] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.702746 4011 consensus_queue.cc:261] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.704427 4011 ts_tablet_manager.cc:1480] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:00.705394 3755 raft_consensus.cc:2490] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:00.705619 4011 tablet_bootstrap.cc:492] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.707401 4011 tablet_bootstrap.cc:654] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.707796 4034 consensus_queue.cc:1049] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:00.712424 4048 consensus_queue.cc:1049] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:00.717813 4011 tablet_bootstrap.cc:492] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.718004 4011 ts_tablet_manager.cc:1449] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20261118 02:55:00.720218 4011 raft_consensus.cc:360] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.720422 4011 raft_consensus.cc:386] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.720497 4011 raft_consensus.cc:748] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.722069 4011 consensus_queue.cc:261] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.722705 3271 catalog_manager.cc:2215] created HMS catalog entry for table [id=790b0571767b49eead05512b819d27db]
I20261118 02:55:00.723526 4012 tablet_bootstrap.cc:492] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.723672 4012 ts_tablet_manager.cc:1449] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.000s
I20261118 02:55:00.724031 4011 ts_tablet_manager.cc:1480] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 02:55:00.724689 4012 raft_consensus.cc:360] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.724864 4012 raft_consensus.cc:386] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.724951 4012 raft_consensus.cc:748] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.725188 4012 consensus_queue.cc:261] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.725708 4011 tablet_bootstrap.cc:492] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.727444 4011 tablet_bootstrap.cc:654] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.730191 3271 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 7) took 1119 ms. Trace:
I20261118 02:55:00.730332 3271 rpcz_store.cc:276] 1118 02:54:59.611194 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:54:59.613665 (+ 2471us) service_pool.cc:224] Handling call
1118 02:55:00.144041 (+530376us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:00.144472 (+ 431us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:00.722655 (+578183us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:00.729824 (+ 7169us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:00.729998 (+ 174us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:00.730141 (+ 143us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:00.724490 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:00.724623 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:00.724631 (+ 8us) write_op.cc:435] Acquired schema lock
1118 02:55:00.724634 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:00.724756 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:00.724771 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:00.724776 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:00.724858 (+ 82us) tablet.cc:709] Row locks acquired
1118 02:55:00.724861 (+ 3us) write_op.cc:260] PREPARE: finished
1118 02:55:00.724904 (+ 43us) write_op.cc:270] Start()
1118 02:55:00.724955 (+ 51us) write_op.cc:276] Timestamp: P: 1794970500724850 usec, L: 0
1118 02:55:00.724958 (+ 3us) op_driver.cc:350] REPLICATION: starting
1118 02:55:00.725147 (+ 189us) log.cc:844] Serialized 432 byte log entry
1118 02:55:00.729130 (+ 3983us) op_driver.cc:466] REPLICATION: finished
1118 02:55:00.729278 (+ 148us) write_op.cc:301] APPLY: starting
1118 02:55:00.729296 (+ 18us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:00.729344 (+ 48us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:00.729373 (+ 29us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:00.729521 (+ 148us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:00.729554 (+ 33us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:00.729560 (+ 6us) write_op.cc:312] APPLY: finished
1118 02:55:00.729633 (+ 73us) log.cc:844] Serialized 28 byte log entry
1118 02:55:00.729695 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:00.729715 (+ 20us) write_op.cc:454] Released schema lock
1118 02:55:00.729722 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:00.729834 (+ 112us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:00.729838 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":462805,"Hive Metastore.run_cpu_time_us":249,"Hive Metastore.run_wall_time_us":114981,"child_traces":[["op",{"apply.queue_time_us":99,"apply.run_cpu_time_us":526,"apply.run_wall_time_us":525,"num_ops":2,"prepare.queue_time_us":1036,"prepare.run_cpu_time_us":770,"prepare.run_wall_time_us":772,"raft.queue_time_us":983,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":235,"replication_time_us":4139}]]}
I20261118 02:55:00.731617 3271 catalog_manager.cc:5840] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: UNKNOWN } } }
I20261118 02:55:00.733893 4012 ts_tablet_manager.cc:1480] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20261118 02:55:00.737948 4011 tablet_bootstrap.cc:492] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.738121 4011 ts_tablet_manager.cc:1449] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20261118 02:55:00.738839 4060 consensus_queue.cc:1049] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:00.740178 4012 tablet_bootstrap.cc:492] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.740470 4011 raft_consensus.cc:360] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.741575 4011 raft_consensus.cc:386] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.741676 4011 raft_consensus.cc:748] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.742358 4011 consensus_queue.cc:261] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.744520 4011 ts_tablet_manager.cc:1480] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20261118 02:55:00.748665 3882 tablet_service.cc:1500] Processing CreateTablet for tablet 55f81e7b62e54ec18c7c13a200edb68f (DEFAULT_TABLE table=default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.748814 3733 tablet_service.cc:1500] Processing CreateTablet for tablet 55f81e7b62e54ec18c7c13a200edb68f (DEFAULT_TABLE table=default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.749074 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55f81e7b62e54ec18c7c13a200edb68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.748943 3271 catalog_manager.cc:5840] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66). New cstate: current_term: 1 leader_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: UNKNOWN } } }
I20261118 02:55:00.749217 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55f81e7b62e54ec18c7c13a200edb68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.753182 3598 tablet_service.cc:1500] Processing CreateTablet for tablet 55f81e7b62e54ec18c7c13a200edb68f (DEFAULT_TABLE table=default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.753626 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55f81e7b62e54ec18c7c13a200edb68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.754141 4040 ts_tablet_manager.cc:1975] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 02:55:00.754325 4040 log.cc:1199] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/9da387134c724364a6c46f65f4e3d605
I20261118 02:55:00.754830 4040 ts_tablet_manager.cc:1996] T 9da387134c724364a6c46f65f4e3d605 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:00.756652 4013 tablet_bootstrap.cc:492] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.756796 3264 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet 9da387134c724364a6c46f65f4e3d605 (table default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]) successfully deleted
I20261118 02:55:00.756966 4013 ts_tablet_manager.cc:1449] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.144s user 0.013s sys 0.029s
I20261118 02:55:00.757735 4013 raft_consensus.cc:360] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.757999 4013 raft_consensus.cc:386] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.758111 4013 raft_consensus.cc:748] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.758447 4013 consensus_queue.cc:261] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.759572 4013 ts_tablet_manager.cc:1480] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20261118 02:55:00.761554 4012 tablet_bootstrap.cc:654] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.761909 4013 tablet_bootstrap.cc:492] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.762741 4011 tablet_bootstrap.cc:492] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.763963 4040 tablet_replica.cc:333] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:00.765817 4011 tablet_bootstrap.cc:654] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.767591 3271 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_87d592a09c3348028719a603956fbce6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:00.768218 4013 tablet_bootstrap.cc:654] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.769277 4040 raft_consensus.cc:2261] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:00.767479 4042 ts_tablet_manager.cc:1975] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 02:55:00.769922 4042 log.cc:1199] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/9da387134c724364a6c46f65f4e3d605
I20261118 02:55:00.770390 4042 ts_tablet_manager.cc:1996] T 9da387134c724364a6c46f65f4e3d605 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:00.770627 4040 raft_consensus.cc:2290] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:00.772243 3263 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet 9da387134c724364a6c46f65f4e3d605 (table default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]) successfully deleted
I20261118 02:55:00.777318 4040 ts_tablet_manager.cc:1962] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:00.780058 4042 tablet_replica.cc:333] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:00.780643 4042 raft_consensus.cc:2261] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:00.781378 4042 raft_consensus.cc:2290] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:00.787606 4042 ts_tablet_manager.cc:1962] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:00.792671 4012 tablet_bootstrap.cc:492] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.792896 4012 ts_tablet_manager.cc:1449] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20261118 02:55:00.793574 4013 tablet_bootstrap.cc:492] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.793830 4013 ts_tablet_manager.cc:1449] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.005s
I20261118 02:55:00.794361 4012 raft_consensus.cc:360] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.794615 4012 raft_consensus.cc:386] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.794698 4012 raft_consensus.cc:748] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.794862 4013 raft_consensus.cc:360] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.795050 4012 consensus_queue.cc:261] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.795126 4013 raft_consensus.cc:386] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.795301 4013 raft_consensus.cc:748] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.795631 4013 consensus_queue.cc:261] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.796227 4013 ts_tablet_manager.cc:1480] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20261118 02:55:00.796656 4013 tablet_bootstrap.cc:492] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.800930 4012 ts_tablet_manager.cc:1480] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20261118 02:55:00.802672 4013 tablet_bootstrap.cc:654] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.803344 4012 tablet_bootstrap.cc:492] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.805401 4042 ts_tablet_manager.cc:1975] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 02:55:00.806660 4042 log.cc:1199] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/d35ba6ef82f9493ca4605294f109b5dc
I20261118 02:55:00.807137 4042 ts_tablet_manager.cc:1996] T d35ba6ef82f9493ca4605294f109b5dc P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:00.809769 4012 tablet_bootstrap.cc:654] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.809237 4011 tablet_bootstrap.cc:492] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.810150 4011 ts_tablet_manager.cc:1449] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.047s user 0.016s sys 0.012s
I20261118 02:55:00.811256 3263 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet d35ba6ef82f9493ca4605294f109b5dc (table default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6]) successfully deleted
I20261118 02:55:00.811589 4011 raft_consensus.cc:360] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.811866 4011 raft_consensus.cc:386] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.811971 4011 raft_consensus.cc:748] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.812325 4011 consensus_queue.cc:261] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.816203 4011 ts_tablet_manager.cc:1480] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20261118 02:55:00.824523 4040 ts_tablet_manager.cc:1975] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 02:55:00.824781 4040 log.cc:1199] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/d35ba6ef82f9493ca4605294f109b5dc
I20261118 02:55:00.825371 4040 ts_tablet_manager.cc:1996] T d35ba6ef82f9493ca4605294f109b5dc P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:00.825892 4012 tablet_bootstrap.cc:492] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.826097 4012 ts_tablet_manager.cc:1449] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20261118 02:55:00.826701 3264 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet d35ba6ef82f9493ca4605294f109b5dc (table default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6]) successfully deleted
I20261118 02:55:00.826996 4012 raft_consensus.cc:360] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.827170 4012 raft_consensus.cc:386] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.827260 4012 raft_consensus.cc:748] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.827485 4012 consensus_queue.cc:261] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.829828 4012 ts_tablet_manager.cc:1480] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 02:55:00.830137 4012 tablet_bootstrap.cc:492] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.831933 4012 tablet_bootstrap.cc:654] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.836763 4013 tablet_bootstrap.cc:492] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.837028 4013 ts_tablet_manager.cc:1449] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.010s
I20261118 02:55:00.837990 4013 raft_consensus.cc:360] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.838212 4013 raft_consensus.cc:386] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.838152 4012 tablet_bootstrap.cc:492] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.838316 4013 raft_consensus.cc:748] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.838516 4012 ts_tablet_manager.cc:1449] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20261118 02:55:00.838630 4013 consensus_queue.cc:261] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.840740 4012 raft_consensus.cc:360] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.841025 4012 raft_consensus.cc:386] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.841150 4012 raft_consensus.cc:748] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.841655 4012 consensus_queue.cc:261] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.841759 4013 ts_tablet_manager.cc:1480] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 02:55:00.844245 4013 tablet_bootstrap.cc:492] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.845513 4012 ts_tablet_manager.cc:1480] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20261118 02:55:00.846978 4013 tablet_bootstrap.cc:654] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.852008 4013 tablet_bootstrap.cc:492] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.852231 4013 ts_tablet_manager.cc:1449] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20261118 02:55:00.853000 4013 raft_consensus.cc:360] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.853535 3275 catalog_manager.cc:2215] created HMS catalog entry for table [id=da0cb25ccfed426ca61f0dd9f93d30ec]
I20261118 02:55:00.853636 4013 raft_consensus.cc:386] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.853713 4013 raft_consensus.cc:748] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.853961 4013 consensus_queue.cc:261] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.854451 4013 ts_tablet_manager.cc:1480] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20261118 02:55:00.864554 3275 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 3) took 1255 ms. Trace:
I20261118 02:55:00.864714 3275 rpcz_store.cc:276] 1118 02:54:59.609056 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:54:59.609636 (+ 580us) service_pool.cc:224] Handling call
1118 02:54:59.615092 (+ 5456us) spinlock_profiling.cc:245] Waited 5.12 ms on lock 0x616000043354. stack: 00007f9c4434a3a6 00007f9c4434a160 00007f9c4333b34d 00007f9c5a90e704 00007f9c5a90e6cd 00007f9c5386082d 00007f9c53860923 00007f9c503ca221 00007f9c582bf50d 00007f9c5844e873 00007f9c4ed57526 00007f9c4ed57255 00007f9c4b4928de 00007f9c4b4917be 00007f9c4b4979f8 00007f9c4b49835f
1118 02:55:00.144955 (+529863us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:00.145346 (+ 391us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:00.853469 (+708123us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:00.864251 (+ 10782us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:00.864379 (+ 128us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:00.864500 (+ 121us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:00.855867 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:00.856008 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:00.856014 (+ 6us) write_op.cc:435] Acquired schema lock
1118 02:55:00.856017 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:00.856141 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:00.856157 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:00.856162 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:00.856248 (+ 86us) tablet.cc:709] Row locks acquired
1118 02:55:00.856250 (+ 2us) write_op.cc:260] PREPARE: finished
1118 02:55:00.856293 (+ 43us) write_op.cc:270] Start()
1118 02:55:00.856329 (+ 36us) write_op.cc:276] Timestamp: P: 1794970500856213 usec, L: 0
1118 02:55:00.856333 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:00.856542 (+ 209us) log.cc:844] Serialized 432 byte log entry
1118 02:55:00.862786 (+ 6244us) op_driver.cc:466] REPLICATION: finished
1118 02:55:00.862941 (+ 155us) write_op.cc:301] APPLY: starting
1118 02:55:00.862959 (+ 18us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:00.863005 (+ 46us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:00.863055 (+ 50us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:00.863200 (+ 145us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:00.863234 (+ 34us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:00.863242 (+ 8us) write_op.cc:312] APPLY: finished
1118 02:55:00.863352 (+ 110us) log.cc:844] Serialized 28 byte log entry
1118 02:55:00.863417 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:00.863439 (+ 22us) write_op.cc:454] Released schema lock
1118 02:55:00.863445 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:00.863515 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:00.863519 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":576948,"Hive Metastore.run_cpu_time_us":302,"Hive Metastore.run_wall_time_us":130946,"spinlock_wait_cycles":11774208,"child_traces":[["op",{"apply.queue_time_us":104,"apply.run_cpu_time_us":568,"apply.run_wall_time_us":566,"num_ops":2,"prepare.queue_time_us":1650,"prepare.run_cpu_time_us":815,"prepare.run_wall_time_us":815,"raft.queue_time_us":1881,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":262,"replication_time_us":6416}]]}
I20261118 02:55:00.879803 3882 tablet_service.cc:1500] Processing CreateTablet for tablet 8f853e6b923444efa79cfbab2e31540e (DEFAULT_TABLE table=default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.880312 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f853e6b923444efa79cfbab2e31540e. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.880692 3598 tablet_service.cc:1500] Processing CreateTablet for tablet 8f853e6b923444efa79cfbab2e31540e (DEFAULT_TABLE table=default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.881098 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f853e6b923444efa79cfbab2e31540e. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.887063 4012 tablet_bootstrap.cc:492] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.887036 4030 raft_consensus.cc:499] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:00.887053 3733 tablet_service.cc:1500] Processing CreateTablet for tablet 8f853e6b923444efa79cfbab2e31540e (DEFAULT_TABLE table=default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.887221 4030 raft_consensus.cc:521] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.887439 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f853e6b923444efa79cfbab2e31540e. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.888222 4030 leader_election.cc:290] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:00.888231 3275 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_295a3e7e0d1f4e9db7078f6e5a5b9da1" } new_table_name: "default.Table_30f082fbe850451b80d98df75ed1081a" modify_external_catalogs: true
I20261118 02:55:00.888932 4012 tablet_bootstrap.cc:654] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.889937 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4ea7c6ad69d94b06bb0e72468db5c834" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:00.890129 3902 raft_consensus.cc:2490] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:00.890599 3553 leader_election.cc:304] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:00.890880 4030 raft_consensus.cc:2826] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:00.890977 4030 raft_consensus.cc:499] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:00.891053 4030 raft_consensus.cc:3096] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.892345 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4ea7c6ad69d94b06bb0e72468db5c834" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:00.892544 3755 raft_consensus.cc:2490] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:00.893671 4030 raft_consensus.cc:521] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.894742 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4ea7c6ad69d94b06bb0e72468db5c834" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:00.894760 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4ea7c6ad69d94b06bb0e72468db5c834" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:00.894923 3755 raft_consensus.cc:3096] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.894924 3902 raft_consensus.cc:3096] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:00.897768 3902 raft_consensus.cc:2490] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:00.897996 3268 catalog_manager.cc:2215] created HMS catalog entry for table [id=4e5b6eaa70d34daf8a68fc17ec1afb66]
I20261118 02:55:00.900777 3553 leader_election.cc:304] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:00.901073 4017 raft_consensus.cc:2826] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:00.901223 4017 raft_consensus.cc:705] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:00.901485 4017 consensus_queue.cc:238] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:00.904270 3268 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 10) took 1292 ms. Trace:
I20261118 02:55:00.904412 3268 rpcz_store.cc:276] 1118 02:54:59.611561 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:54:59.612034 (+ 473us) service_pool.cc:224] Handling call
1118 02:55:00.145818 (+533784us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:00.146246 (+ 428us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:00.897951 (+751705us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:00.903975 (+ 6024us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:00.904113 (+ 138us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:00.904231 (+ 118us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:00.898727 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:00.898853 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:00.898859 (+ 6us) write_op.cc:435] Acquired schema lock
1118 02:55:00.898861 (+ 2us) tablet.cc:670] Decoding operations
1118 02:55:00.898953 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:00.898966 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:00.898971 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:00.899050 (+ 79us) tablet.cc:709] Row locks acquired
1118 02:55:00.899053 (+ 3us) write_op.cc:260] PREPARE: finished
1118 02:55:00.899091 (+ 38us) write_op.cc:270] Start()
1118 02:55:00.899122 (+ 31us) write_op.cc:276] Timestamp: P: 1794970500899004 usec, L: 0
1118 02:55:00.899125 (+ 3us) op_driver.cc:350] REPLICATION: starting
1118 02:55:00.899307 (+ 182us) log.cc:844] Serialized 432 byte log entry
1118 02:55:00.902517 (+ 3210us) op_driver.cc:466] REPLICATION: finished
1118 02:55:00.902804 (+ 287us) write_op.cc:301] APPLY: starting
1118 02:55:00.902822 (+ 18us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:00.902879 (+ 57us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:00.902913 (+ 34us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:00.903031 (+ 118us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:00.903064 (+ 33us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:00.903071 (+ 7us) write_op.cc:312] APPLY: finished
1118 02:55:00.903146 (+ 75us) log.cc:844] Serialized 28 byte log entry
1118 02:55:00.903203 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:00.903224 (+ 21us) write_op.cc:454] Released schema lock
1118 02:55:00.903230 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:00.903294 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:00.903297 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":707074,"Hive Metastore.run_cpu_time_us":269,"Hive Metastore.run_wall_time_us":42372,"child_traces":[["op",{"apply.queue_time_us":231,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":488,"num_ops":2,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":697,"prepare.run_wall_time_us":698,"raft.queue_time_us":356,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":3360}]]}
I20261118 02:55:00.905124 3268 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_2e3917ff98f2474984580e61caa9c17b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:00.909484 3755 raft_consensus.cc:2490] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:00.909751 4030 leader_election.cc:290] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:00.914669 4013 tablet_bootstrap.cc:492] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.916502 4013 tablet_bootstrap.cc:654] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.918707 3881 tablet_service.cc:1500] Processing CreateTablet for tablet ffe8d4c111d14054b76f9c7724072a94 (DEFAULT_TABLE table=default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.919165 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe8d4c111d14054b76f9c7724072a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.921376 4011 tablet_bootstrap.cc:492] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.923242 4011 tablet_bootstrap.cc:654] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.924502 3734 tablet_service.cc:1500] Processing CreateTablet for tablet ffe8d4c111d14054b76f9c7724072a94 (DEFAULT_TABLE table=default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.924963 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe8d4c111d14054b76f9c7724072a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.928500 3598 tablet_service.cc:1500] Processing CreateTablet for tablet ffe8d4c111d14054b76f9c7724072a94 (DEFAULT_TABLE table=default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:00.928941 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe8d4c111d14054b76f9c7724072a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:00.939474 4030 consensus_queue.cc:1049] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:00.941191 4013 tablet_bootstrap.cc:492] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.941363 4013 ts_tablet_manager.cc:1449] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.007s
I20261118 02:55:00.942764 4013 raft_consensus.cc:360] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.942956 4013 raft_consensus.cc:386] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.943046 4013 raft_consensus.cc:748] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.943265 4013 consensus_queue.cc:261] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.944295 4011 tablet_bootstrap.cc:492] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:00.944451 4011 ts_tablet_manager.cc:1449] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.000s
I20261118 02:55:00.945395 4011 raft_consensus.cc:360] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.945576 4011 raft_consensus.cc:386] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.945667 4011 raft_consensus.cc:748] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.945909 4011 consensus_queue.cc:261] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.947181 4011 ts_tablet_manager.cc:1480] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:00.947479 4011 tablet_bootstrap.cc:492] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:00.949267 4033 consensus_queue.cc:1049] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:00.949764 4012 tablet_bootstrap.cc:492] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:00.949949 4012 ts_tablet_manager.cc:1449] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.018s
I20261118 02:55:00.950810 4012 raft_consensus.cc:360] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.951066 4012 raft_consensus.cc:386] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:00.951202 4012 raft_consensus.cc:748] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:00.951659 4012 consensus_queue.cc:261] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.953037 4012 ts_tablet_manager.cc:1480] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20261118 02:55:00.953457 4013 ts_tablet_manager.cc:1480] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20261118 02:55:00.953749 4013 tablet_bootstrap.cc:492] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:00.955120 4012 tablet_bootstrap.cc:492] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:00.956234 4011 tablet_bootstrap.cc:654] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.958338 4013 tablet_bootstrap.cc:654] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.959293 4012 tablet_bootstrap.cc:654] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:00.976816 4013 tablet_bootstrap.cc:492] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:00.977128 4019 raft_consensus.cc:499] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:00.977315 4019 raft_consensus.cc:521] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:00.979369 4019 leader_election.cc:290] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:00.997922 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "55f81e7b62e54ec18c7c13a200edb68f" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:00.998242 3617 raft_consensus.cc:2490] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:00.999222 3838 leader_election.cc:304] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:01.000617 4019 raft_consensus.cc:2826] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:01.000756 4019 raft_consensus.cc:499] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:01.000834 4019 raft_consensus.cc:3096] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.009092 4019 raft_consensus.cc:521] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.010766 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "55f81e7b62e54ec18c7c13a200edb68f" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:01.011016 3617 raft_consensus.cc:3096] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.014350 3617 raft_consensus.cc:2490] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:01.015102 3838 leader_election.cc:304] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:01.015421 3269 catalog_manager.cc:2215] created HMS catalog entry for table [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]
I20261118 02:55:01.022222 4019 leader_election.cc:290] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:01.022687 3269 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 9) took 1411 ms. Trace:
I20261118 02:55:01.022745 4019 raft_consensus.cc:2826] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:01.022899 4019 raft_consensus.cc:705] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:01.022835 3269 rpcz_store.cc:276] 1118 02:54:59.611425 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:54:59.614026 (+ 2601us) service_pool.cc:224] Handling call
1118 02:55:00.146766 (+532740us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:00.147148 (+ 382us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:01.015370 (+868222us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:01.022294 (+ 6924us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:01.022512 (+ 218us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:01.022640 (+ 128us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:01.016315 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:01.016469 (+ 154us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:01.016476 (+ 7us) write_op.cc:435] Acquired schema lock
1118 02:55:01.016479 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:01.016584 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:01.016599 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:01.016605 (+ 6us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:01.016695 (+ 90us) tablet.cc:709] Row locks acquired
1118 02:55:01.016697 (+ 2us) write_op.cc:260] PREPARE: finished
1118 02:55:01.016738 (+ 41us) write_op.cc:270] Start()
1118 02:55:01.016776 (+ 38us) write_op.cc:276] Timestamp: P: 1794970501016627 usec, L: 0
1118 02:55:01.016780 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:01.016985 (+ 205us) log.cc:844] Serialized 432 byte log entry
1118 02:55:01.020815 (+ 3830us) op_driver.cc:466] REPLICATION: finished
1118 02:55:01.020951 (+ 136us) write_op.cc:301] APPLY: starting
1118 02:55:01.020970 (+ 19us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:01.021019 (+ 49us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:01.021060 (+ 41us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:01.021188 (+ 128us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:01.021224 (+ 36us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:01.021232 (+ 8us) write_op.cc:312] APPLY: finished
1118 02:55:01.021315 (+ 83us) log.cc:844] Serialized 28 byte log entry
1118 02:55:01.021380 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:01.021403 (+ 23us) write_op.cc:454] Released schema lock
1118 02:55:01.021410 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:01.021479 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:01.021483 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":748571,"Hive Metastore.run_cpu_time_us":292,"Hive Metastore.run_wall_time_us":118994,"child_traces":[["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":519,"apply.run_wall_time_us":519,"num_ops":2,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":792,"prepare.run_wall_time_us":793,"raft.queue_time_us":453,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":308,"replication_time_us":3997}]]}
I20261118 02:55:01.023607 4019 consensus_queue.cc:238] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.034076 3598 tablet_service.cc:1500] Processing CreateTablet for tablet 5f05a8e5b21945a5bd3df86a5467a615 (DEFAULT_TABLE table=default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:01.034158 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 5f05a8e5b21945a5bd3df86a5467a615 (DEFAULT_TABLE table=default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:01.034607 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f05a8e5b21945a5bd3df86a5467a615. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:01.034626 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f05a8e5b21945a5bd3df86a5467a615. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:01.040361 4013 ts_tablet_manager.cc:1449] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.003s
I20261118 02:55:01.041601 3269 catalog_manager.cc:5840] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: UNKNOWN } } }
I20261118 02:55:01.042656 4011 tablet_bootstrap.cc:492] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:01.042830 4011 ts_tablet_manager.cc:1449] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.000s
I20261118 02:55:01.043771 4011 raft_consensus.cc:360] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.043972 4011 raft_consensus.cc:386] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:01.044456 4011 raft_consensus.cc:748] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:01.044795 4011 consensus_queue.cc:261] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.048317 4013 raft_consensus.cc:360] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.048539 4013 raft_consensus.cc:386] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:01.048625 4013 raft_consensus.cc:748] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:01.048975 4013 consensus_queue.cc:261] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.050591 4013 ts_tablet_manager.cc:1480] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20261118 02:55:01.051680 4013 tablet_bootstrap.cc:492] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:01.053499 4013 tablet_bootstrap.cc:654] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:01.066936 4017 raft_consensus.cc:499] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:01.068428 4017 raft_consensus.cc:521] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.070628 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "9a19cb906c974009aa9df99a14a78024" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:01.070953 3755 raft_consensus.cc:2490] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:01.071683 3552 leader_election.cc:304] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:01.071919 4087 raft_consensus.cc:499] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:01.071976 4030 raft_consensus.cc:2826] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:01.072077 4030 raft_consensus.cc:499] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:01.072219 4030 raft_consensus.cc:3096] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.072235 4087 raft_consensus.cc:521] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.073030 4087 leader_election.cc:290] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:01.073980 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "9a19cb906c974009aa9df99a14a78024" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:01.074193 3902 raft_consensus.cc:2490] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:01.074751 4017 leader_election.cc:290] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:01.075047 4030 raft_consensus.cc:521] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.076458 3269 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_bca0059ea2004174b89866f3e8cc68fb" } modify_external_catalogs: true
I20261118 02:55:01.076751 3269 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_bca0059ea2004174b89866f3e8cc68fb" } modify_external_catalogs: true
I20261118 02:55:01.077927 4030 leader_election.cc:290] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:01.078564 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "9a19cb906c974009aa9df99a14a78024" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:01.078768 3902 raft_consensus.cc:3096] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.081962 3902 raft_consensus.cc:2490] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:01.083328 3553 leader_election.cc:304] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:01.084998 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "9a19cb906c974009aa9df99a14a78024" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:01.085238 3755 raft_consensus.cc:3096] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.088471 3755 raft_consensus.cc:2490] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:01.089386 4030 raft_consensus.cc:2826] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:01.089607 4030 raft_consensus.cc:705] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:01.089985 4030 consensus_queue.cc:238] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.104441 4017 consensus_queue.cc:1049] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.109282 4017 consensus_queue.cc:1049] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 02:55:01.120455 3838 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.229.65:33705, user_credentials={real_user=slave}} blocked reactor thread for 105128us
I20261118 02:55:01.121537 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8f853e6b923444efa79cfbab2e31540e" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:01.121726 3618 raft_consensus.cc:2490] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:01.122246 4058 raft_consensus.cc:499] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:01.122401 4058 raft_consensus.cc:521] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.123129 4058 leader_election.cc:290] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:01.123740 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "55f81e7b62e54ec18c7c13a200edb68f" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:01.123821 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "55f81e7b62e54ec18c7c13a200edb68f" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:01.124086 3618 raft_consensus.cc:2415] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99a39b1d9d814deeb3a8a644e9b392ef in current term 1: Already voted for candidate 3f1973986ec1466eab5bec01dd81654f in this term.
I20261118 02:55:01.124585 3838 leader_election.cc:304] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:01.124856 4083 consensus_queue.cc:1049] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.125450 4083 raft_consensus.cc:2826] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:01.125617 4083 raft_consensus.cc:499] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:01.125764 4083 raft_consensus.cc:3096] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.129177 4083 raft_consensus.cc:521] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.129601 3691 leader_election.cc:304] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f
I20261118 02:55:01.129951 4058 raft_consensus.cc:3096] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.133100 4058 raft_consensus.cc:2771] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 02:55:01.134994 4012 tablet_bootstrap.cc:492] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:01.135224 4012 ts_tablet_manager.cc:1449] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.180s user 0.019s sys 0.103s
I20261118 02:55:01.136164 4012 raft_consensus.cc:360] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.136442 4012 raft_consensus.cc:386] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:01.136569 4012 raft_consensus.cc:748] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:01.136917 4012 consensus_queue.cc:261] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.137537 4012 ts_tablet_manager.cc:1480] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20261118 02:55:01.137873 4012 tablet_bootstrap.cc:492] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:01.139792 4012 tablet_bootstrap.cc:654] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:01.150574 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "55f81e7b62e54ec18c7c13a200edb68f" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:01.150856 3755 raft_consensus.cc:2490] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:01.151506 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "55f81e7b62e54ec18c7c13a200edb68f" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:01.154713 3755 raft_consensus.cc:2490] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:01.156157 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8f853e6b923444efa79cfbab2e31540e" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:01.156404 3755 raft_consensus.cc:2490] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:01.157614 4083 leader_election.cc:290] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:01.140549 4011 ts_tablet_manager.cc:1480] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.098s user 0.002s sys 0.001s
I20261118 02:55:01.159271 4013 tablet_bootstrap.cc:492] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:01.159497 4083 consensus_queue.cc:1049] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.160163 4013 ts_tablet_manager.cc:1449] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.000s
I20261118 02:55:01.161015 4013 raft_consensus.cc:360] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.158180 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8f853e6b923444efa79cfbab2e31540e" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:01.161988 3617 raft_consensus.cc:3096] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.162770 4013 raft_consensus.cc:386] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:01.162910 4013 raft_consensus.cc:748] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:01.163129 4013 consensus_queue.cc:261] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.164708 3617 raft_consensus.cc:2490] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:01.165182 3838 leader_election.cc:304] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:01.165723 4083 raft_consensus.cc:2826] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:01.166100 4083 raft_consensus.cc:705] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:01.166388 4083 consensus_queue.cc:238] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.168346 4013 ts_tablet_manager.cc:1480] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20261118 02:55:01.169687 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8f853e6b923444efa79cfbab2e31540e" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:01.169888 3755 raft_consensus.cc:3096] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.174764 3755 raft_consensus.cc:2490] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
W20261118 02:55:01.181120 3263 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.229.124:36695, user_credentials={real_user=slave}} blocked reactor thread for 208234us
I20261118 02:55:01.186060 3881 tablet_service.cc:1500] Processing CreateTablet for tablet 5f05a8e5b21945a5bd3df86a5467a615 (DEFAULT_TABLE table=default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:01.186553 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f05a8e5b21945a5bd3df86a5467a615. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:01.193660 4011 tablet_bootstrap.cc:492] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:01.195525 4011 tablet_bootstrap.cc:654] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:01.213032 4011 tablet_bootstrap.cc:492] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:01.213302 4011 ts_tablet_manager.cc:1449] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.008s
I20261118 02:55:01.213562 4030 raft_consensus.cc:499] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:01.213743 4030 raft_consensus.cc:521] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.214416 4030 leader_election.cc:290] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:01.215081 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "6bf520b76802491da8e3b090ad2152da" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:01.215914 3902 raft_consensus.cc:2490] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:01.216032 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "6bf520b76802491da8e3b090ad2152da" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:01.216260 3755 raft_consensus.cc:2490] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:01.216509 3553 leader_election.cc:304] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:01.217114 4030 raft_consensus.cc:2826] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:01.217226 4030 raft_consensus.cc:499] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:01.217320 4030 raft_consensus.cc:3096] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.220525 4030 raft_consensus.cc:521] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.222060 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "6bf520b76802491da8e3b090ad2152da" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:01.222258 3902 raft_consensus.cc:3096] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.225169 3902 raft_consensus.cc:2490] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:01.225698 3553 leader_election.cc:304] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:01.226019 4017 raft_consensus.cc:2826] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:01.226150 4017 raft_consensus.cc:705] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:01.226444 4017 consensus_queue.cc:238] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.229610 4030 leader_election.cc:290] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:01.231071 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "6bf520b76802491da8e3b090ad2152da" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:01.231716 3755 raft_consensus.cc:3096] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.232752 4011 raft_consensus.cc:360] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.235993 3755 raft_consensus.cc:2490] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:01.241838 4017 consensus_queue.cc:1049] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.244262 4011 raft_consensus.cc:386] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:01.244421 4011 raft_consensus.cc:748] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:01.244746 4011 consensus_queue.cc:261] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.246165 4011 ts_tablet_manager.cc:1480] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20261118 02:55:01.249260 4012 tablet_bootstrap.cc:492] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:01.249439 4012 ts_tablet_manager.cc:1449] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.112s user 0.010s sys 0.016s
I20261118 02:55:01.250572 4012 raft_consensus.cc:360] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.250659 4030 consensus_queue.cc:1049] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.251631 4012 raft_consensus.cc:386] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:01.251731 4012 raft_consensus.cc:748] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:01.251968 4012 consensus_queue.cc:261] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.254069 4012 ts_tablet_manager.cc:1480] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 02:55:01.261523 4019 consensus_queue.cc:1049] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.273392 4019 consensus_queue.cc:1049] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.286320 4019 raft_consensus.cc:499] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:01.286552 4019 raft_consensus.cc:521] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.287194 4019 leader_election.cc:290] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:01.288276 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "5f05a8e5b21945a5bd3df86a5467a615" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:01.288504 3617 raft_consensus.cc:2490] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:01.290966 3838 leader_election.cc:304] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:01.291399 4087 raft_consensus.cc:2826] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:01.291510 4087 raft_consensus.cc:499] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:01.291585 4087 raft_consensus.cc:3096] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.294031 4087 raft_consensus.cc:521] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.294570 4087 leader_election.cc:290] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:01.299003 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "5f05a8e5b21945a5bd3df86a5467a615" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:01.299207 3752 raft_consensus.cc:3096] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.301959 3752 raft_consensus.cc:2490] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:01.305069 4058 raft_consensus.cc:499] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:01.305225 4058 raft_consensus.cc:521] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.305819 4058 leader_election.cc:290] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:01.306428 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ffe8d4c111d14054b76f9c7724072a94" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:01.306638 3902 raft_consensus.cc:2490] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:01.307621 3690 leader_election.cc:304] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:01.307973 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ffe8d4c111d14054b76f9c7724072a94" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:01.308045 3836 leader_election.cc:304] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:01.308195 3617 raft_consensus.cc:2490] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:01.308547 4083 raft_consensus.cc:2826] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:01.308665 4083 raft_consensus.cc:705] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:01.308970 4083 consensus_queue.cc:238] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:01.309405 4058 raft_consensus.cc:2826] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:01.309492 4058 raft_consensus.cc:499] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:01.309582 4058 raft_consensus.cc:3096] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.302659 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "5f05a8e5b21945a5bd3df86a5467a615" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:01.313270 3755 raft_consensus.cc:2398] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1. Re-sending same reply.
I20261118 02:55:01.315402 4058 raft_consensus.cc:521] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.316243 4058 leader_election.cc:290] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:01.316540 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ffe8d4c111d14054b76f9c7724072a94" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:01.316818 3902 raft_consensus.cc:3096] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.317504 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ffe8d4c111d14054b76f9c7724072a94" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:01.317711 3617 raft_consensus.cc:3096] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.320818 3617 raft_consensus.cc:2490] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:01.321260 3691 leader_election.cc:304] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:01.321542 4058 raft_consensus.cc:2826] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:01.321683 4058 raft_consensus.cc:705] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:01.321924 4058 consensus_queue.cc:238] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:01.325083 3902 raft_consensus.cc:2490] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:01.329159 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "5f05a8e5b21945a5bd3df86a5467a615" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:01.330018 3618 raft_consensus.cc:3096] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:01.333078 4083 consensus_queue.cc:1049] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.337407 3618 raft_consensus.cc:2490] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:01.347039 4083 consensus_queue.cc:1049] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.350419 4105 consensus_queue.cc:1049] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261118 02:55:01.351373 4058 consensus_queue.cc:1049] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:01.592870 4110 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:55:01.593219 4110 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:01.594197 4110 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:01.594687 3287 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
W20261118 02:55:01.628887 4047 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:55:01.629298 4047 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:55:01.629408 4047 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:55:01.630364 3333 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:01.630635 3333 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:01.630990 4110 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20261118 02:55:01.643215 4047 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:55:01.643329 4047 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:55:01.643406 4047 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:55:01.643468 4047 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:55:01.660548 4047 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.124:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.229.124
--webserver_port=39557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:55:01.660801 3277 catalog_manager.cc:3543] renamed table default.table_dc3979f8c8b148caa37b0a8ef420ba9c [id=052f3134115244a6a2443e96121313e6] in HMS: new name default.table_8ae376d4d4d64414b5c08dc3c2e58bc4
I20261118 02:55:01.662734 4047 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:55:01.665131 4047 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:55:01.677729 3271 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_87d592a09c3348028719a603956fbce6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:01.678263 3270 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a14cb6414555446286bf35bc5c6fb910 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:01.677757 3273 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_22f31dd1ff9e4185a6618f11b5493434 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:01.679179 3268 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2e3917ff98f2474984580e61caa9c17b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:01.691787 4117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:01.693372 4118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:55:01.698589 4047 server_base.cc:1061] running on GCE node
W20261118 02:55:01.699322 4121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:55:01.700428 4047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:55:01.716599 4047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:55:01.718187 4047 hybrid_clock.cc:648] HybridClock initialized: now 1794970501718094 us; error 95 us; skew 500 ppm
I20261118 02:55:01.718766 4047 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:55:01.752295 3277 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:52040 (request call id 11) took 1628 ms. Trace:
I20261118 02:55:01.752509 3277 rpcz_store.cc:276] 1118 02:55:00.123548 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:00.123601 (+ 53us) service_pool.cc:224] Handling call
1118 02:55:01.064934 (+941333us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:01.751542 (+686608us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:01.751876 (+ 334us) catalog_manager.cc:3766] Apply alter schema
1118 02:55:01.752251 (+ 375us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":7953,"Hive Metastore.run_cpu_time_us":1145,"Hive Metastore.run_wall_time_us":587486}
I20261118 02:55:01.753866 3277 catalog_manager.cc:5840] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20261118 02:55:01.767570 3277 catalog_manager.cc:5840] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: UNKNOWN } } }
I20261118 02:55:01.778620 4047 webserver.cc:577] Webserver started at http://127.2.229.124:39557/ using document root <none> and password file <none>
I20261118 02:55:01.779771 4047 fs_manager.cc:362] Metadata directory not provided
I20261118 02:55:01.779938 4047 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:55:01.796772 4047 fs_manager.cc:714] Time spent opening directory manager: real 0.014s user 0.002s sys 0.005s
I20261118 02:55:01.806308 4133 tablet.cc:1759] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:01.814383 4132 tablet.cc:1759] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:01.816519 4131 tablet.cc:1759] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:01.819792 4136 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:55:01.821978 4047 fs_manager.cc:730] Time spent opening block manager: real 0.021s user 0.003s sys 0.001s
I20261118 02:55:01.822213 4047 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
uuid: "813a6937cea442ef81645fccc32207f4"
format_stamp: "Formatted at 2026-11-18 02:54:57 on dist-test-slave-49p1"
I20261118 02:55:01.823239 4047 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20261118 02:55:01.835572 3263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20261118 02:55:01.844447 3277 catalog_manager.cc:5840] T ffe8d4c111d14054b76f9c7724072a94 P 3f1973986ec1466eab5bec01dd81654f reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66). New cstate: current_term: 1 leader_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } }
I20261118 02:55:01.845209 3277 catalog_manager.cc:5840] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: HEALTHY } } }
I20261118 02:55:01.845793 3277 catalog_manager.cc:5840] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: HEALTHY } } }
I20261118 02:55:01.846341 3277 catalog_manager.cc:5840] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } }
I20261118 02:55:01.860460 4047 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:55:01.862219 4047 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:55:01.862593 4047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:55:01.865468 3277 catalog_manager.cc:6393] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6] alter complete (version 1)
I20261118 02:55:01.871469 3277 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_571d1360811c42dc815efb439b2b52e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:01.884214 3275 catalog_manager.cc:3543] renamed table default.table_295a3e7e0d1f4e9db7078f6e5a5b9da1 [id=44b390866720455b8d2714eae004383b] in HMS: new name default.table_30f082fbe850451b80d98df75ed1081a
I20261118 02:55:01.954710 4187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.124:36695 every 8 connection(s)
I20261118 02:55:01.954723 4047 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.124:36695
I20261118 02:55:01.957198 4047 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
I20261118 02:55:01.964733 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 4047
I20261118 02:55:01.965569 3274 catalog_manager.cc:3543] renamed table default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7] in HMS: new name default.table_4f7796d5d5d54def843b971e66d15ea6
I20261118 02:55:01.968478 4188 sys_catalog.cc:263] Verifying existing consensus state
I20261118 02:55:01.972883 4188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap starting.
I20261118 02:55:01.985036 4188 log.cc:826] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:02.005002 4188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:02.005785 4188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap complete.
I20261118 02:55:02.014441 4188 raft_consensus.cc:360] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:02.015486 4188 raft_consensus.cc:748] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813a6937cea442ef81645fccc32207f4, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.016491 4188 consensus_queue.cc:261] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:02.018721 4192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:02.019734 4192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:02.021673 4188 sys_catalog.cc:565] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:55:02.045423 3272 catalog_manager.cc:3543] renamed table default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da] in HMS: new name default.table_a7f3710d39fb447da94e16d4ab3933fd
I20261118 02:55:02.052615 4207 catalog_manager.cc:820] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20261118 02:55:02.097942 3271 catalog_manager.cc:2215] created HMS catalog entry for table [id=9724ed64fa5043bcbfee1caefac8e86d]
I20261118 02:55:02.104640 3271 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 22) took 1660 ms. Trace:
I20261118 02:55:02.104787 3271 rpcz_store.cc:276] 1118 02:55:00.444349 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:00.767401 (+323052us) service_pool.cc:224] Handling call
1118 02:55:01.677904 (+910503us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:01.678444 (+ 540us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:02.097894 (+419450us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:02.104342 (+ 6448us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:02.104455 (+ 113us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:02.104570 (+ 115us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:02.098761 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:02.098904 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:02.098911 (+ 7us) write_op.cc:435] Acquired schema lock
1118 02:55:02.098914 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:02.099018 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:02.099033 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:02.099040 (+ 7us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:02.099138 (+ 98us) tablet.cc:709] Row locks acquired
1118 02:55:02.099140 (+ 2us) write_op.cc:260] PREPARE: finished
1118 02:55:02.099183 (+ 43us) write_op.cc:270] Start()
1118 02:55:02.099220 (+ 37us) write_op.cc:276] Timestamp: P: 1794970502099070 usec, L: 0
1118 02:55:02.099224 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:02.099443 (+ 219us) log.cc:844] Serialized 432 byte log entry
1118 02:55:02.103542 (+ 4099us) op_driver.cc:466] REPLICATION: finished
1118 02:55:02.103675 (+ 133us) write_op.cc:301] APPLY: starting
1118 02:55:02.103692 (+ 17us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:02.103738 (+ 46us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:02.103778 (+ 40us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:02.103899 (+ 121us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:02.103933 (+ 34us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:02.103941 (+ 8us) write_op.cc:312] APPLY: finished
1118 02:55:02.104021 (+ 80us) log.cc:844] Serialized 28 byte log entry
1118 02:55:02.104087 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.104142 (+ 55us) write_op.cc:454] Released schema lock
1118 02:55:02.104149 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:02.104225 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.104229 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":366798,"Hive Metastore.run_cpu_time_us":356,"Hive Metastore.run_wall_time_us":52256,"child_traces":[["op",{"apply.queue_time_us":85,"apply.run_cpu_time_us":540,"apply.run_wall_time_us":539,"num_ops":2,"prepare.queue_time_us":76,"prepare.run_cpu_time_us":810,"prepare.run_wall_time_us":811,"raft.queue_time_us":1095,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":320,"replication_time_us":4288}]]}
I20261118 02:55:02.105614 3271 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_a4f08da750c54c49b9b7d73ebcf52624" } modify_external_catalogs: true
I20261118 02:55:02.105814 3271 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_a4f08da750c54c49b9b7d73ebcf52624" } modify_external_catalogs: true
I20261118 02:55:02.118363 3598 tablet_service.cc:1500] Processing CreateTablet for tablet c92c29c30ab6411dabe6f02331d29b7a (DEFAULT_TABLE table=default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.118593 3881 tablet_service.cc:1500] Processing CreateTablet for tablet c92c29c30ab6411dabe6f02331d29b7a (DEFAULT_TABLE table=default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.118872 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c92c29c30ab6411dabe6f02331d29b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.118957 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c92c29c30ab6411dabe6f02331d29b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.125578 3734 tablet_service.cc:1500] Processing CreateTablet for tablet c92c29c30ab6411dabe6f02331d29b7a (DEFAULT_TABLE table=default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.126056 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c92c29c30ab6411dabe6f02331d29b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.149720 4213 tablet_bootstrap.cc:492] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:02.151823 4213 tablet_bootstrap.cc:654] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.153157 4212 tablet_bootstrap.cc:492] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:02.155045 4212 tablet_bootstrap.cc:654] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.165388 4213 tablet_bootstrap.cc:492] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:02.165678 4213 ts_tablet_manager.cc:1449] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20261118 02:55:02.166639 4213 raft_consensus.cc:360] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.167119 4213 raft_consensus.cc:386] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.167273 4213 raft_consensus.cc:748] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.167644 4213 consensus_queue.cc:261] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.168357 4213 ts_tablet_manager.cc:1480] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:02.175135 4212 tablet_bootstrap.cc:492] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:02.175374 4212 ts_tablet_manager.cc:1449] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20261118 02:55:02.176163 4211 tablet_bootstrap.cc:492] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:02.176326 4212 raft_consensus.cc:360] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.176578 4212 raft_consensus.cc:386] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.176705 4212 raft_consensus.cc:748] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.177008 4212 consensus_queue.cc:261] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.177603 4212 ts_tablet_manager.cc:1480] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:02.179162 4211 tablet_bootstrap.cc:654] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.183950 3948 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:55:02.188684 4153 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" instance_seqno: 1794970499527172) as {username='slave'} at 127.2.229.67:52477; Asking this server to re-register.
I20261118 02:55:02.195184 3948 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:02.198149 4153 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:02.206529 3801 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:55:02.209568 4153 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" instance_seqno: 1794970499028671) as {username='slave'} at 127.2.229.66:50711; Asking this server to re-register.
I20261118 02:55:02.209733 4211 tablet_bootstrap.cc:492] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:02.209895 4211 ts_tablet_manager.cc:1449] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.013s
I20261118 02:55:02.210455 3801 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:02.212265 4153 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:02.213104 4211 raft_consensus.cc:360] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.213397 4211 raft_consensus.cc:386] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.213542 4211 raft_consensus.cc:748] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.215049 4211 consensus_queue.cc:261] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.219259 4211 ts_tablet_manager.cc:1480] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20261118 02:55:02.224360 3664 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:55:02.226665 4153 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" instance_seqno: 1794970498532688) as {username='slave'} at 127.2.229.65:46439; Asking this server to re-register.
I20261118 02:55:02.236189 3664 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:02.238484 3276 catalog_manager.cc:3543] renamed table default.table_df2a47bb353f4cdcb4c16399a99cb722 [id=8a5ea4b95bee4db7af9ba875250d564b] in HMS: new name default.table_0385cef6593c419eb426657c8cbef2bf
I20261118 02:55:02.239441 4153 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:02.277581 3270 catalog_manager.cc:2215] created HMS catalog entry for table [id=0ce698410f004edaa9ad4ba1afe2de73]
I20261118 02:55:02.283869 3270 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 21) took 1867 ms. Trace:
I20261118 02:55:02.284021 3270 rpcz_store.cc:276] 1118 02:55:00.416429 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:00.648272 (+231843us) service_pool.cc:224] Handling call
1118 02:55:01.678371 (+1030099us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:01.678780 (+ 409us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:02.277527 (+598747us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:02.283561 (+ 6034us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:02.283695 (+ 134us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:02.283825 (+ 130us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:02.278391 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:02.278506 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:02.278512 (+ 6us) write_op.cc:435] Acquired schema lock
1118 02:55:02.278515 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:02.278602 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:02.278614 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:02.278619 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:02.278694 (+ 75us) tablet.cc:709] Row locks acquired
1118 02:55:02.278696 (+ 2us) write_op.cc:260] PREPARE: finished
1118 02:55:02.278739 (+ 43us) write_op.cc:270] Start()
1118 02:55:02.278771 (+ 32us) write_op.cc:276] Timestamp: P: 1794970502278689 usec, L: 0
1118 02:55:02.278775 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:02.278967 (+ 192us) log.cc:844] Serialized 432 byte log entry
1118 02:55:02.282695 (+ 3728us) op_driver.cc:466] REPLICATION: finished
1118 02:55:02.282822 (+ 127us) write_op.cc:301] APPLY: starting
1118 02:55:02.282840 (+ 18us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:02.282881 (+ 41us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:02.282907 (+ 26us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:02.283006 (+ 99us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:02.283039 (+ 33us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:02.283046 (+ 7us) write_op.cc:312] APPLY: finished
1118 02:55:02.283118 (+ 72us) log.cc:844] Serialized 28 byte log entry
1118 02:55:02.283185 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.283204 (+ 19us) write_op.cc:454] Released schema lock
1118 02:55:02.283211 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:02.283280 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.283283 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":559564,"Hive Metastore.run_cpu_time_us":291,"Hive Metastore.run_wall_time_us":38786,"child_traces":[["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":451,"apply.run_wall_time_us":451,"num_ops":2,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":678,"raft.queue_time_us":489,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":271,"replication_time_us":3890}]]}
I20261118 02:55:02.295192 4192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:02.295473 4192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:02.296294 4126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:02.296555 4126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:02.311007 3598 tablet_service.cc:1500] Processing CreateTablet for tablet 65bd6fcaed5b47c3a2478d375ace9636 (DEFAULT_TABLE table=default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.311528 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bd6fcaed5b47c3a2478d375ace9636. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.311496 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 65bd6fcaed5b47c3a2478d375ace9636 (DEFAULT_TABLE table=default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.311944 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bd6fcaed5b47c3a2478d375ace9636. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.314611 3273 catalog_manager.cc:2215] created HMS catalog entry for table [id=aea45564ce10469393d3c7cc03b0bb9d]
I20261118 02:55:02.315302 3881 tablet_service.cc:1500] Processing CreateTablet for tablet 65bd6fcaed5b47c3a2478d375ace9636 (DEFAULT_TABLE table=default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.315774 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bd6fcaed5b47c3a2478d375ace9636. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.318720 4211 tablet_bootstrap.cc:492] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:02.320504 4211 tablet_bootstrap.cc:654] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.323490 3270 master_service.cc:965] ReplaceTablet: received request to replace tablet 6bf520b76802491da8e3b090ad2152da from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:02.329592 4212 tablet_bootstrap.cc:492] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:02.331429 4212 tablet_bootstrap.cc:654] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.333096 3273 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 14) took 2009 ms. Trace:
I20261118 02:55:02.333614 4213 tablet_bootstrap.cc:492] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:02.333204 3273 rpcz_store.cc:276] 1118 02:55:00.323551 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:00.349401 (+ 25850us) service_pool.cc:224] Handling call
1118 02:55:01.678955 (+1329554us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:01.679415 (+ 460us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:02.314569 (+635154us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:02.332799 (+ 18230us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:02.332918 (+ 119us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:02.333036 (+ 118us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:02.316651 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:02.316811 (+ 160us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:02.316817 (+ 6us) write_op.cc:435] Acquired schema lock
1118 02:55:02.316820 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:02.316960 (+ 140us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:02.316974 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:02.316979 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:02.317060 (+ 81us) tablet.cc:709] Row locks acquired
1118 02:55:02.317063 (+ 3us) write_op.cc:260] PREPARE: finished
1118 02:55:02.317103 (+ 40us) write_op.cc:270] Start()
1118 02:55:02.317140 (+ 37us) write_op.cc:276] Timestamp: P: 1794970502317046 usec, L: 0
1118 02:55:02.317144 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:02.317347 (+ 203us) log.cc:844] Serialized 432 byte log entry
1118 02:55:02.329320 (+ 11973us) op_driver.cc:466] REPLICATION: finished
1118 02:55:02.332185 (+ 2865us) write_op.cc:301] APPLY: starting
1118 02:55:02.332202 (+ 17us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:02.332240 (+ 38us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:02.332274 (+ 34us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:02.332383 (+ 109us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:02.332415 (+ 32us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:02.332423 (+ 8us) write_op.cc:312] APPLY: finished
1118 02:55:02.332495 (+ 72us) log.cc:844] Serialized 28 byte log entry
1118 02:55:02.332555 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.332573 (+ 18us) write_op.cc:454] Released schema lock
1118 02:55:02.332580 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:02.332648 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.332652 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":597749,"Hive Metastore.run_cpu_time_us":280,"Hive Metastore.run_wall_time_us":37187,"child_traces":[["op",{"apply.queue_time_us":2794,"apply.run_cpu_time_us":463,"apply.run_wall_time_us":464,"num_ops":2,"prepare.queue_time_us":1283,"prepare.run_cpu_time_us":781,"prepare.run_wall_time_us":781,"replication_time_us":12146}]]}
I20261118 02:55:02.335309 4213 tablet_bootstrap.cc:654] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.343427 3273 master_service.cc:965] ReplaceTablet: received request to replace tablet 4ea7c6ad69d94b06bb0e72468db5c834 from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:02.352924 3270 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet 6bf520b76802491da8e3b090ad2152da
I20261118 02:55:02.354514 3270 catalog_manager.cc:6936] ReplaceTablet: tablet 6bf520b76802491da8e3b090ad2152da deleted and replaced by tablet 2de409a48f5b488484d74008f186d166
I20261118 02:55:02.354751 3270 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:52040 (request call id 28) took 1612 ms. Trace:
I20261118 02:55:02.355038 3598 tablet_service.cc:1547] Processing DeleteTablet for tablet 6bf520b76802491da8e3b090ad2152da with delete_type TABLET_DATA_DELETED (replaced by tablet 2de409a48f5b488484d74008f186d166) from {username='slave'} at 127.0.0.1:52990
I20261118 02:55:02.355044 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet 6bf520b76802491da8e3b090ad2152da with delete_type TABLET_DATA_DELETED (replaced by tablet 2de409a48f5b488484d74008f186d166) from {username='slave'} at 127.0.0.1:50234
I20261118 02:55:02.355412 3270 rpcz_store.cc:276] 1118 02:55:00.742530 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:02.323403 (+1580873us) service_pool.cc:224] Handling call
1118 02:55:02.354695 (+ 31292us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:02.338524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:02.338693 (+ 169us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:02.338700 (+ 7us) write_op.cc:435] Acquired schema lock
1118 02:55:02.338703 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:02.338833 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:02.338848 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:02.338854 (+ 6us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:02.338939 (+ 85us) tablet.cc:709] Row locks acquired
1118 02:55:02.338942 (+ 3us) write_op.cc:260] PREPARE: finished
1118 02:55:02.338985 (+ 43us) write_op.cc:270] Start()
1118 02:55:02.339024 (+ 39us) write_op.cc:276] Timestamp: P: 1794970502338925 usec, L: 0
1118 02:55:02.339028 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:02.339278 (+ 250us) log.cc:844] Serialized 658 byte log entry
1118 02:55:02.351807 (+ 12529us) op_driver.cc:466] REPLICATION: finished
1118 02:55:02.351905 (+ 98us) write_op.cc:301] APPLY: starting
1118 02:55:02.351921 (+ 16us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:02.351959 (+ 38us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:02.351987 (+ 28us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:02.352083 (+ 96us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:02.352161 (+ 78us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1118 02:55:02.352170 (+ 9us) write_op.cc:312] APPLY: finished
1118 02:55:02.352241 (+ 71us) log.cc:844] Serialized 28 byte log entry
1118 02:55:02.352309 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.352329 (+ 20us) write_op.cc:454] Released schema lock
1118 02:55:02.352336 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:02.352405 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.352409 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":491,"apply.run_wall_time_us":490,"num_ops":2,"prepare.queue_time_us":4341,"prepare.run_cpu_time_us":843,"prepare.run_wall_time_us":843,"replication_time_us":12745}]]}
I20261118 02:55:02.356783 4229 tablet_replica.cc:333] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:02.359972 4229 raft_consensus.cc:2261] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:02.360608 4229 raft_consensus.cc:2290] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.360976 3734 tablet_service.cc:1547] Processing DeleteTablet for tablet 6bf520b76802491da8e3b090ad2152da with delete_type TABLET_DATA_DELETED (replaced by tablet 2de409a48f5b488484d74008f186d166) from {username='slave'} at 127.0.0.1:57880
I20261118 02:55:02.361258 4229 ts_tablet_manager.cc:1962] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.362175 3268 catalog_manager.cc:2215] created HMS catalog entry for table [id=2584489d44ac4c6b90ca7c9f390c617e]
I20261118 02:55:02.367789 4230 tablet_replica.cc:333] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:02.368572 4230 raft_consensus.cc:2261] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:02.368898 3273 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet 4ea7c6ad69d94b06bb0e72468db5c834
I20261118 02:55:02.369431 4231 tablet_replica.cc:333] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:02.369632 3598 tablet_service.cc:1547] Processing DeleteTablet for tablet 4ea7c6ad69d94b06bb0e72468db5c834 with delete_type TABLET_DATA_DELETED (replaced by tablet e71c5cd71ebf4fa6bd158d59ed2b191d) from {username='slave'} at 127.0.0.1:52990
I20261118 02:55:02.370877 4231 raft_consensus.cc:2261] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:02.371163 4231 raft_consensus.cc:2290] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.371469 4230 raft_consensus.cc:2290] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.372083 4230 ts_tablet_manager.cc:1962] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.373778 3881 tablet_service.cc:1500] Processing CreateTablet for tablet a013d180f2cd430ab1ba30b39fd27591 (DEFAULT_TABLE table=default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.374286 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a013d180f2cd430ab1ba30b39fd27591. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.377398 3598 tablet_service.cc:1500] Processing CreateTablet for tablet a013d180f2cd430ab1ba30b39fd27591 (DEFAULT_TABLE table=default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.377952 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a013d180f2cd430ab1ba30b39fd27591. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.379402 4211 tablet_bootstrap.cc:492] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:02.384011 3268 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 24) took 1831 ms. Trace:
I20261118 02:55:02.384150 3268 rpcz_store.cc:276] 1118 02:55:00.552748 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:00.904912 (+352164us) service_pool.cc:224] Handling call
1118 02:55:01.679266 (+774354us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:01.679660 (+ 394us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:02.362137 (+682477us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:02.383726 (+ 21589us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:02.383838 (+ 112us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:02.383940 (+ 102us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:02.368143 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:02.368266 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:02.368271 (+ 5us) write_op.cc:435] Acquired schema lock
1118 02:55:02.368273 (+ 2us) tablet.cc:670] Decoding operations
1118 02:55:02.368354 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:02.368363 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:02.368368 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:02.368433 (+ 65us) tablet.cc:709] Row locks acquired
1118 02:55:02.368436 (+ 3us) write_op.cc:260] PREPARE: finished
1118 02:55:02.368464 (+ 28us) write_op.cc:270] Start()
1118 02:55:02.368497 (+ 33us) write_op.cc:276] Timestamp: P: 1794970502368402 usec, L: 0
1118 02:55:02.368500 (+ 3us) op_driver.cc:350] REPLICATION: starting
1118 02:55:02.378467 (+ 9967us) log.cc:844] Serialized 432 byte log entry
1118 02:55:02.378752 (+ 285us) spinlock_profiling.cc:245] Waited 9.57 ms on lock 0x616000043354. stack: 00007f9c4434a3a6 00007f9c4434a160 00007f9c4333b34d 00007f9c5a90e704 00007f9c5a90e6cd 00007f9c5a90e09a 00007f9c5039c632 00007f9c517c0935 00007f9c517bf459 00007f9c517c7880 00007f9c517c768d 0000000000372c7c 00007f9c4439283f 00007f9c44395b3b 00007f9c443959b8 0000000000372c7c
1118 02:55:02.382673 (+ 3921us) op_driver.cc:466] REPLICATION: finished
1118 02:55:02.383283 (+ 610us) write_op.cc:301] APPLY: starting
1118 02:55:02.383299 (+ 16us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:02.383341 (+ 42us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:02.383344 (+ 3us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:02.383448 (+ 104us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:02.383480 (+ 32us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:02.383487 (+ 7us) write_op.cc:312] APPLY: finished
1118 02:55:02.383559 (+ 72us) log.cc:844] Serialized 28 byte log entry
1118 02:55:02.383622 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.383641 (+ 19us) write_op.cc:454] Released schema lock
1118 02:55:02.383648 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:02.383722 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.383726 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":634760,"Hive Metastore.run_cpu_time_us":274,"Hive Metastore.run_wall_time_us":47388,"child_traces":[["op",{"apply.queue_time_us":548,"apply.run_cpu_time_us":437,"apply.run_wall_time_us":437,"num_ops":2,"prepare.queue_time_us":900,"prepare.run_cpu_time_us":1105,"prepare.run_wall_time_us":10678,"raft.queue_time_us":779,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":298,"replication_time_us":14144,"spinlock_wait_cycles":22015360}]]}
I20261118 02:55:02.384773 3268 master_service.cc:965] ReplaceTablet: received request to replace tablet 4c230b673b8f487fbc79bdfb11a48628 from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:02.386333 3734 tablet_service.cc:1500] Processing CreateTablet for tablet a013d180f2cd430ab1ba30b39fd27591 (DEFAULT_TABLE table=default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.393581 4211 ts_tablet_manager.cc:1449] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.075s user 0.009s sys 0.005s
I20261118 02:55:02.394498 4211 raft_consensus.cc:360] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.394727 4211 raft_consensus.cc:386] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.394809 4211 raft_consensus.cc:748] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.396087 4230 ts_tablet_manager.cc:1975] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:02.396301 4230 log.cc:1199] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/6bf520b76802491da8e3b090ad2152da
I20261118 02:55:02.396831 4230 ts_tablet_manager.cc:1996] T 6bf520b76802491da8e3b090ad2152da P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:02.398039 3263 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet 6bf520b76802491da8e3b090ad2152da (table default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]) successfully deleted
I20261118 02:55:02.400060 3268 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet 4c230b673b8f487fbc79bdfb11a48628
I20261118 02:55:02.400735 3268 catalog_manager.cc:6936] ReplaceTablet: tablet 4c230b673b8f487fbc79bdfb11a48628 deleted and replaced by tablet 78189cccd457466bb5d01231038fec24
I20261118 02:55:02.400923 3268 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:52040 (request call id 31) took 1542 ms. Trace:
I20261118 02:55:02.401161 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet 4c230b673b8f487fbc79bdfb11a48628 with delete_type TABLET_DATA_DELETED (replaced by tablet 78189cccd457466bb5d01231038fec24) from {username='slave'} at 127.0.0.1:50234
I20261118 02:55:02.401871 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet 4c230b673b8f487fbc79bdfb11a48628 with delete_type TABLET_DATA_DELETED (replaced by tablet 78189cccd457466bb5d01231038fec24) from {username='slave'} at 127.0.0.1:52990
I20261118 02:55:02.401073 3268 rpcz_store.cc:276] 1118 02:55:00.858542 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:02.384690 (+1526148us) service_pool.cc:224] Handling call
1118 02:55:02.392370 (+ 7680us) spinlock_profiling.cc:245] Waited 7.29 ms on lock 0x616000043354. stack: 00007f9c4434a3a6 00007f9c4434a160 00007f9c4333b34d 00007f9c5a90e704 00007f9c5a90e6cd 00007f9c5386082d 00007f9c53860923 00007f9c503ca221 00007f9c582bf50d 00007f9c5845b6fb 00007f9c4ed63e67 00007f9c4ed63b95 00007f9c4b4928de 00007f9c4b4917be 00007f9c4b4979f8 00007f9c4b49835f
1118 02:55:02.400892 (+ 8522us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:02.395389 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:02.395504 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:02.395509 (+ 5us) write_op.cc:435] Acquired schema lock
1118 02:55:02.395511 (+ 2us) tablet.cc:670] Decoding operations
1118 02:55:02.395609 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:02.395634 (+ 25us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:02.395639 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:02.395711 (+ 72us) tablet.cc:709] Row locks acquired
1118 02:55:02.395714 (+ 3us) write_op.cc:260] PREPARE: finished
1118 02:55:02.395752 (+ 38us) write_op.cc:270] Start()
1118 02:55:02.395782 (+ 30us) write_op.cc:276] Timestamp: P: 1794970502395682 usec, L: 0
1118 02:55:02.395786 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:02.396010 (+ 224us) log.cc:844] Serialized 658 byte log entry
1118 02:55:02.398940 (+ 2930us) op_driver.cc:466] REPLICATION: finished
1118 02:55:02.399075 (+ 135us) write_op.cc:301] APPLY: starting
1118 02:55:02.399094 (+ 19us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:02.399142 (+ 48us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:02.399210 (+ 68us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:02.399346 (+ 136us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:02.399401 (+ 55us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1118 02:55:02.399409 (+ 8us) write_op.cc:312] APPLY: finished
1118 02:55:02.399480 (+ 71us) log.cc:844] Serialized 28 byte log entry
1118 02:55:02.399726 (+ 246us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.399746 (+ 20us) write_op.cc:454] Released schema lock
1118 02:55:02.399753 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:02.399819 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.399823 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":16767360,"child_traces":[["op",{"apply.queue_time_us":38,"apply.run_cpu_time_us":582,"apply.run_wall_time_us":745,"num_ops":2,"prepare.queue_time_us":2070,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":722,"raft.queue_time_us":354,"replication_time_us":3124}]]}
I20261118 02:55:02.402199 4230 tablet_replica.cc:333] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:02.402442 4230 raft_consensus.cc:2261] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:02.402681 4230 raft_consensus.cc:2290] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.403153 3268 master_service.cc:965] ReplaceTablet: received request to replace tablet 4c230b673b8f487fbc79bdfb11a48628 from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:02.403334 4230 ts_tablet_manager.cc:1962] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.403766 3268 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:52040 (request call id 32) took 1543 ms. Trace:
I20261118 02:55:02.403903 3268 rpcz_store.cc:276] 1118 02:55:00.860437 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:02.403080 (+1542643us) service_pool.cc:224] Handling call
1118 02:55:02.403713 (+ 633us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261118 02:55:02.404774 3268 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_c113c2b57e17479e92e241234a241af7" } modify_external_catalogs: true
I20261118 02:55:02.405263 3268 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_c113c2b57e17479e92e241234a241af7" } modify_external_catalogs: true
I20261118 02:55:02.405880 4211 consensus_queue.cc:261] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.406497 4211 ts_tablet_manager.cc:1480] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20261118 02:55:02.409996 4230 ts_tablet_manager.cc:1975] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:02.410462 3270 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "default.Table_51d36ff687ca47b5995e5693d7b8931b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:02.371889 4231 ts_tablet_manager.cc:1962] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.412957 3273 catalog_manager.cc:6936] ReplaceTablet: tablet 4ea7c6ad69d94b06bb0e72468db5c834 deleted and replaced by tablet e71c5cd71ebf4fa6bd158d59ed2b191d
I20261118 02:55:02.413075 3273 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:52040 (request call id 29) took 1639 ms. Trace:
I20261118 02:55:02.413159 3273 rpcz_store.cc:276] 1118 02:55:00.773826 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:02.343341 (+1569515us) service_pool.cc:224] Handling call
1118 02:55:02.368602 (+ 25261us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.368608 (+ 6us) write_op.cc:454] Released schema lock
1118 02:55:02.413044 (+ 44436us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:02.361862 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:02.361976 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:02.361981 (+ 5us) write_op.cc:435] Acquired schema lock
1118 02:55:02.361984 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:02.362084 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:02.362095 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:02.362100 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:02.362169 (+ 69us) tablet.cc:709] Row locks acquired
1118 02:55:02.362172 (+ 3us) write_op.cc:260] PREPARE: finished
1118 02:55:02.362207 (+ 35us) write_op.cc:270] Start()
1118 02:55:02.362237 (+ 30us) write_op.cc:276] Timestamp: P: 1794970502362144 usec, L: 0
1118 02:55:02.362241 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:02.362420 (+ 179us) log.cc:844] Serialized 658 byte log entry
1118 02:55:02.365341 (+ 2921us) op_driver.cc:466] REPLICATION: finished
1118 02:55:02.365435 (+ 94us) write_op.cc:301] APPLY: starting
1118 02:55:02.365447 (+ 12us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:02.365491 (+ 44us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:02.365516 (+ 25us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:02.365605 (+ 89us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:02.365667 (+ 62us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1118 02:55:02.365674 (+ 7us) write_op.cc:312] APPLY: finished
1118 02:55:02.365737 (+ 63us) log.cc:844] Serialized 28 byte log entry
1118 02:55:02.365795 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.365811 (+ 16us) write_op.cc:454] Released schema lock
1118 02:55:02.365817 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":430,"num_ops":2,"prepare.queue_time_us":17231,"prepare.run_cpu_time_us":663,"prepare.run_wall_time_us":663,"raft.queue_time_us":1191,"raft.run_cpu_time_us":563,"raft.run_wall_time_us":564,"replication_time_us":3075}]]}
I20261118 02:55:02.414131 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a013d180f2cd430ab1ba30b39fd27591. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.408886 3733 tablet_service.cc:1547] Processing DeleteTablet for tablet 4c230b673b8f487fbc79bdfb11a48628 with delete_type TABLET_DATA_DELETED (replaced by tablet 78189cccd457466bb5d01231038fec24) from {username='slave'} at 127.0.0.1:57880
I20261118 02:55:02.412323 3735 tablet_service.cc:1547] Processing DeleteTablet for tablet 4ea7c6ad69d94b06bb0e72468db5c834 with delete_type TABLET_DATA_DELETED (replaced by tablet e71c5cd71ebf4fa6bd158d59ed2b191d) from {username='slave'} at 127.0.0.1:57880
I20261118 02:55:02.417141 3597 tablet_service.cc:1500] Processing CreateTablet for tablet 2de409a48f5b488484d74008f186d166 (DEFAULT_TABLE table=default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.411877 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet 4ea7c6ad69d94b06bb0e72468db5c834 with delete_type TABLET_DATA_DELETED (replaced by tablet e71c5cd71ebf4fa6bd158d59ed2b191d) from {username='slave'} at 127.0.0.1:50234
I20261118 02:55:02.417624 4230 log.cc:1199] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/4c230b673b8f487fbc79bdfb11a48628
I20261118 02:55:02.418151 4230 ts_tablet_manager.cc:1996] T 4c230b673b8f487fbc79bdfb11a48628 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:02.420329 3735 tablet_service.cc:1500] Processing CreateTablet for tablet 2de409a48f5b488484d74008f186d166 (DEFAULT_TABLE table=default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.420812 3735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de409a48f5b488484d74008f186d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.421273 3881 tablet_service.cc:1500] Processing CreateTablet for tablet 2de409a48f5b488484d74008f186d166 (DEFAULT_TABLE table=default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.423233 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de409a48f5b488484d74008f186d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.423996 4229 ts_tablet_manager.cc:1975] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:02.424155 4229 log.cc:1199] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/6bf520b76802491da8e3b090ad2152da
I20261118 02:55:02.424561 4229 ts_tablet_manager.cc:1996] T 6bf520b76802491da8e3b090ad2152da P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:02.427963 4231 ts_tablet_manager.cc:1975] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:02.428192 4231 log.cc:1199] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/6bf520b76802491da8e3b090ad2152da
I20261118 02:55:02.428731 4231 ts_tablet_manager.cc:1996] T 6bf520b76802491da8e3b090ad2152da P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:02.429561 3596 tablet_service.cc:1500] Processing CreateTablet for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (DEFAULT_TABLE table=default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.429872 3262 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet 6bf520b76802491da8e3b090ad2152da (table default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]) successfully deleted
I20261118 02:55:02.430226 4231 tablet_replica.cc:333] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:02.431022 3882 tablet_service.cc:1500] Processing CreateTablet for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (DEFAULT_TABLE table=default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.431501 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e71c5cd71ebf4fa6bd158d59ed2b191d. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.431843 4231 raft_consensus.cc:2261] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:02.432245 3273 master_service.cc:965] ReplaceTablet: received request to replace tablet 9a19cb906c974009aa9df99a14a78024 from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:02.432397 4231 raft_consensus.cc:2290] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.432947 4231 ts_tablet_manager.cc:1962] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.431545 3735 tablet_service.cc:1500] Processing CreateTablet for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (DEFAULT_TABLE table=default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.435521 3735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e71c5cd71ebf4fa6bd158d59ed2b191d. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.439710 4213 tablet_bootstrap.cc:492] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:02.439857 3273 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet 9a19cb906c974009aa9df99a14a78024
I20261118 02:55:02.439952 4213 ts_tablet_manager.cc:1449] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.015s
I20261118 02:55:02.440919 3595 tablet_service.cc:1547] Processing DeleteTablet for tablet 9a19cb906c974009aa9df99a14a78024 with delete_type TABLET_DATA_DELETED (replaced by tablet e9323addc63245bfacb3854782425347) from {username='slave'} at 127.0.0.1:52990
I20261118 02:55:02.441011 4213 raft_consensus.cc:360] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.441272 4213 raft_consensus.cc:386] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.441354 4213 raft_consensus.cc:748] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.441845 3263 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet 4c230b673b8f487fbc79bdfb11a48628 (table default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]) successfully deleted
I20261118 02:55:02.442184 4230 tablet_replica.cc:333] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:02.442389 4213 consensus_queue.cc:261] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.442389 4230 raft_consensus.cc:2261] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:02.443663 3733 tablet_service.cc:1547] Processing DeleteTablet for tablet 9a19cb906c974009aa9df99a14a78024 with delete_type TABLET_DATA_DELETED (replaced by tablet e9323addc63245bfacb3854782425347) from {username='slave'} at 127.0.0.1:57880
I20261118 02:55:02.444335 4230 raft_consensus.cc:2290] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.444782 4213 ts_tablet_manager.cc:1480] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 02:55:02.445071 4213 tablet_bootstrap.cc:492] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:02.449146 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de409a48f5b488484d74008f186d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.450366 4213 tablet_bootstrap.cc:654] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.452953 3596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e71c5cd71ebf4fa6bd158d59ed2b191d. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.461302 4229 tablet_replica.cc:333] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:02.461537 4229 raft_consensus.cc:2261] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:02.461993 4229 raft_consensus.cc:2290] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.462831 4229 ts_tablet_manager.cc:1962] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.463047 4213 tablet_bootstrap.cc:492] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:02.463263 4213 ts_tablet_manager.cc:1449] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20261118 02:55:02.464167 4213 raft_consensus.cc:360] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:02.464414 4213 raft_consensus.cc:386] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.464530 4213 raft_consensus.cc:748] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.464838 4213 consensus_queue.cc:261] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:02.466874 4213 ts_tablet_manager.cc:1480] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20261118 02:55:02.467221 4213 tablet_bootstrap.cc:492] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:02.468192 3264 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet 6bf520b76802491da8e3b090ad2152da (table default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]) successfully deleted
I20261118 02:55:02.468778 4211 tablet_bootstrap.cc:492] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:02.470914 4213 tablet_bootstrap.cc:654] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.475835 4230 ts_tablet_manager.cc:1962] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.477862 4231 ts_tablet_manager.cc:1975] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:02.478031 4231 log.cc:1199] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/4c230b673b8f487fbc79bdfb11a48628
I20261118 02:55:02.479214 4213 tablet_bootstrap.cc:492] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:02.479290 4211 tablet_bootstrap.cc:654] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.479466 4213 ts_tablet_manager.cc:1449] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20261118 02:55:02.480432 4213 raft_consensus.cc:360] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.480701 4213 raft_consensus.cc:386] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.481022 4213 raft_consensus.cc:748] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.481406 4213 consensus_queue.cc:261] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.483502 4213 ts_tablet_manager.cc:1480] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 02:55:02.483843 4213 tablet_bootstrap.cc:492] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:02.487402 4231 ts_tablet_manager.cc:1996] T 4c230b673b8f487fbc79bdfb11a48628 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:02.487635 4213 tablet_bootstrap.cc:654] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.488178 4212 tablet_bootstrap.cc:492] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:02.488426 4212 ts_tablet_manager.cc:1449] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.159s user 0.005s sys 0.004s
I20261118 02:55:02.488587 3262 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet 4c230b673b8f487fbc79bdfb11a48628 (table default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]) successfully deleted
I20261118 02:55:02.490651 4212 raft_consensus.cc:360] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.490962 4212 raft_consensus.cc:386] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.491072 4212 raft_consensus.cc:748] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.491425 4212 consensus_queue.cc:261] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.492177 4212 ts_tablet_manager.cc:1480] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 02:55:02.492547 4212 tablet_bootstrap.cc:492] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:02.494400 4212 tablet_bootstrap.cc:654] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.497686 4229 ts_tablet_manager.cc:1975] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:02.499058 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet 9a19cb906c974009aa9df99a14a78024 with delete_type TABLET_DATA_DELETED (replaced by tablet e9323addc63245bfacb3854782425347) from {username='slave'} at 127.0.0.1:50234
I20261118 02:55:02.499738 4213 tablet_bootstrap.cc:492] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:02.499893 4213 ts_tablet_manager.cc:1449] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20261118 02:55:02.500203 3273 catalog_manager.cc:6936] ReplaceTablet: tablet 9a19cb906c974009aa9df99a14a78024 deleted and replaced by tablet e9323addc63245bfacb3854782425347
I20261118 02:55:02.500412 3273 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:52040 (request call id 34) took 1629 ms. Trace:
I20261118 02:55:02.500705 4213 raft_consensus.cc:360] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.500890 4213 raft_consensus.cc:386] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.500978 4213 raft_consensus.cc:748] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.501272 4213 consensus_queue.cc:261] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.502074 4229 log.cc:1199] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/4ea7c6ad69d94b06bb0e72468db5c834
I20261118 02:55:02.500546 3273 rpcz_store.cc:276] 1118 02:55:00.871173 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:02.432174 (+1561001us) service_pool.cc:224] Handling call
1118 02:55:02.500360 (+ 68186us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:02.434678 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:02.434809 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:02.434816 (+ 7us) write_op.cc:435] Acquired schema lock
1118 02:55:02.434819 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:02.434930 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:02.434942 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:02.434947 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:02.435016 (+ 69us) tablet.cc:709] Row locks acquired
1118 02:55:02.435018 (+ 2us) write_op.cc:260] PREPARE: finished
1118 02:55:02.435064 (+ 46us) write_op.cc:270] Start()
1118 02:55:02.435099 (+ 35us) write_op.cc:276] Timestamp: P: 1794970502434982 usec, L: 0
1118 02:55:02.435103 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:02.435386 (+ 283us) log.cc:844] Serialized 658 byte log entry
1118 02:55:02.438512 (+ 3126us) op_driver.cc:466] REPLICATION: finished
1118 02:55:02.438730 (+ 218us) write_op.cc:301] APPLY: starting
1118 02:55:02.438747 (+ 17us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:02.438794 (+ 47us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:02.438819 (+ 25us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:02.438915 (+ 96us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:02.438972 (+ 57us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1118 02:55:02.438980 (+ 8us) write_op.cc:312] APPLY: finished
1118 02:55:02.439049 (+ 69us) log.cc:844] Serialized 28 byte log entry
1118 02:55:02.439111 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.439129 (+ 18us) write_op.cc:454] Released schema lock
1118 02:55:02.439135 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:02.439204 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:02.439208 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":147,"apply.run_cpu_time_us":481,"apply.run_wall_time_us":481,"num_ops":2,"prepare.queue_time_us":1327,"prepare.run_cpu_time_us":797,"prepare.run_wall_time_us":841,"raft.queue_time_us":1891,"raft.run_cpu_time_us":675,"raft.run_wall_time_us":1988,"replication_time_us":3377,"spinlock_wait_cycles":66048}]]}
I20261118 02:55:02.503659 4213 ts_tablet_manager.cc:1480] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 02:55:02.507390 4229 ts_tablet_manager.cc:1996] T 4ea7c6ad69d94b06bb0e72468db5c834 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:02.508754 4230 ts_tablet_manager.cc:1975] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:02.509831 4229 tablet_replica.cc:333] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:02.510353 3264 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet 4ea7c6ad69d94b06bb0e72468db5c834 (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]) successfully deleted
I20261118 02:55:02.494484 4231 tablet_replica.cc:333] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:02.510623 4230 log.cc:1199] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/4ea7c6ad69d94b06bb0e72468db5c834
I20261118 02:55:02.510756 4231 raft_consensus.cc:2261] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:02.511085 4231 raft_consensus.cc:2290] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.512200 4231 ts_tablet_manager.cc:1962] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.512450 4230 ts_tablet_manager.cc:1996] T 4ea7c6ad69d94b06bb0e72468db5c834 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:02.512244 4229 raft_consensus.cc:2261] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:02.513509 4230 tablet_replica.cc:333] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:02.513728 4229 raft_consensus.cc:2290] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.514289 4229 ts_tablet_manager.cc:1962] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.514379 3263 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet 4ea7c6ad69d94b06bb0e72468db5c834 (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]) successfully deleted
I20261118 02:55:02.515167 4230 raft_consensus.cc:2261] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:02.515419 4230 raft_consensus.cc:2290] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.517704 4230 ts_tablet_manager.cc:1962] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.518064 4215 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:02.518205 4215 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.518909 4215 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:02.520200 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c92c29c30ab6411dabe6f02331d29b7a" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:02.520404 3755 raft_consensus.cc:2490] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:02.521052 3836 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:02.521936 4215 raft_consensus.cc:2826] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:02.522042 4215 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:02.521989 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c92c29c30ab6411dabe6f02331d29b7a" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:02.522117 4215 raft_consensus.cc:3096] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.522203 3618 raft_consensus.cc:2490] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:02.523371 4212 tablet_bootstrap.cc:492] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:02.523517 4212 ts_tablet_manager.cc:1449] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20261118 02:55:02.524334 4212 raft_consensus.cc:360] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.524521 4212 raft_consensus.cc:386] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.524616 4212 raft_consensus.cc:748] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.524766 4215 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.524919 4212 consensus_queue.cc:261] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.526166 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c92c29c30ab6411dabe6f02331d29b7a" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:02.526369 3618 raft_consensus.cc:3096] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.526746 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c92c29c30ab6411dabe6f02331d29b7a" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:02.526921 3755 raft_consensus.cc:3096] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.527249 4231 ts_tablet_manager.cc:1975] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:02.527379 4231 log.cc:1199] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/4ea7c6ad69d94b06bb0e72468db5c834
I20261118 02:55:02.527817 4231 ts_tablet_manager.cc:1996] T 4ea7c6ad69d94b06bb0e72468db5c834 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:02.528297 4212 ts_tablet_manager.cc:1480] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 02:55:02.528940 3262 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet 4ea7c6ad69d94b06bb0e72468db5c834 (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]) successfully deleted
I20261118 02:55:02.529165 3618 raft_consensus.cc:2490] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:02.530246 3755 raft_consensus.cc:2490] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:02.530700 3836 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:02.530949 4212 tablet_bootstrap.cc:492] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:02.531028 4210 raft_consensus.cc:2826] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:02.531577 4229 ts_tablet_manager.cc:1975] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:02.531693 4229 log.cc:1199] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/4c230b673b8f487fbc79bdfb11a48628
I20261118 02:55:02.532657 4231 tablet_replica.cc:333] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:02.532696 4229 ts_tablet_manager.cc:1996] T 4c230b673b8f487fbc79bdfb11a48628 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:02.532874 4231 raft_consensus.cc:2261] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:02.533051 4231 raft_consensus.cc:2290] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.533205 4212 tablet_bootstrap.cc:654] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.533658 4229 tablet_replica.cc:333] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:02.533648 4231 ts_tablet_manager.cc:1962] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.533855 4229 raft_consensus.cc:2261] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:02.534322 3264 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet 4c230b673b8f487fbc79bdfb11a48628 (table default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]) successfully deleted
I20261118 02:55:02.535019 4229 raft_consensus.cc:2290] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:02.536906 4215 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:02.536726 4229 ts_tablet_manager.cc:1962] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:02.501953 4211 tablet_bootstrap.cc:492] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:02.540230 4211 ts_tablet_manager.cc:1449] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.072s user 0.001s sys 0.003s
I20261118 02:55:02.540940 4211 raft_consensus.cc:360] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.541126 4211 raft_consensus.cc:386] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.541237 4211 raft_consensus.cc:748] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.541453 4211 consensus_queue.cc:261] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.546059 3273 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_287099a68a3540f1be0284c71d4d7056" } modify_external_catalogs: true
I20261118 02:55:02.548307 3273 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "default.Table_287099a68a3540f1be0284c71d4d7056" } modify_external_catalogs: true
I20261118 02:55:02.548563 4210 raft_consensus.cc:705] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:02.548883 4210 consensus_queue.cc:238] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.552723 3735 tablet_service.cc:1500] Processing CreateTablet for tablet 489beea71ae74236addee59c153cfee7 (DEFAULT_TABLE table=default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.553171 3735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489beea71ae74236addee59c153cfee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.553092 3881 tablet_service.cc:1500] Processing CreateTablet for tablet 489beea71ae74236addee59c153cfee7 (DEFAULT_TABLE table=default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.553480 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489beea71ae74236addee59c153cfee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.556941 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 78189cccd457466bb5d01231038fec24 (DEFAULT_TABLE table=default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.557351 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78189cccd457466bb5d01231038fec24. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.558972 3882 tablet_service.cc:1500] Processing CreateTablet for tablet 78189cccd457466bb5d01231038fec24 (DEFAULT_TABLE table=default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.559351 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78189cccd457466bb5d01231038fec24. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.559295 3596 tablet_service.cc:1500] Processing CreateTablet for tablet 78189cccd457466bb5d01231038fec24 (DEFAULT_TABLE table=default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.559686 3596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78189cccd457466bb5d01231038fec24. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.563303 4213 tablet_bootstrap.cc:492] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:02.565783 4213 tablet_bootstrap.cc:654] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.542007 4211 ts_tablet_manager.cc:1480] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 02:55:02.573877 4239 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:02.575912 4239 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:02.577234 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a013d180f2cd430ab1ba30b39fd27591" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
W20261118 02:55:02.578177 3836 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Illegal state: must be running to vote when last-logged opid is not known
I20261118 02:55:02.548470 4229 ts_tablet_manager.cc:1975] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:02.579119 4229 log.cc:1199] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/9a19cb906c974009aa9df99a14a78024
I20261118 02:55:02.552182 3598 tablet_service.cc:1500] Processing CreateTablet for tablet 489beea71ae74236addee59c153cfee7 (DEFAULT_TABLE table=default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.579300 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a013d180f2cd430ab1ba30b39fd27591" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:02.579530 4229 ts_tablet_manager.cc:1996] T 9a19cb906c974009aa9df99a14a78024 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:02.579659 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489beea71ae74236addee59c153cfee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.583029 4231 ts_tablet_manager.cc:1975] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:02.583178 4231 log.cc:1199] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/9a19cb906c974009aa9df99a14a78024
I20261118 02:55:02.583501 4231 ts_tablet_manager.cc:1996] T 9a19cb906c974009aa9df99a14a78024 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
W20261118 02:55:02.583515 3838 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Illegal state: must be running to vote when last-logged opid is not known
I20261118 02:55:02.583633 3838 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:02.584122 3264 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet 9a19cb906c974009aa9df99a14a78024 (table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]) successfully deleted
I20261118 02:55:02.586706 4215 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 02:55:02.588336 4239 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:02.593148 4212 tablet_bootstrap.cc:492] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:02.593330 4212 ts_tablet_manager.cc:1449] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.005s
I20261118 02:55:02.594102 4212 raft_consensus.cc:360] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:02.594316 4212 raft_consensus.cc:386] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.594388 4212 raft_consensus.cc:748] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.594635 4212 consensus_queue.cc:261] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:02.596073 4212 ts_tablet_manager.cc:1480] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20261118 02:55:02.596426 4212 tablet_bootstrap.cc:492] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:02.596057 3262 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet 9a19cb906c974009aa9df99a14a78024 (table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]) successfully deleted
I20261118 02:55:02.600968 4212 tablet_bootstrap.cc:654] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.616334 4211 tablet_bootstrap.cc:492] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:02.617105 4212 tablet_bootstrap.cc:492] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:02.617264 4212 ts_tablet_manager.cc:1449] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20261118 02:55:02.618093 4212 raft_consensus.cc:360] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.618280 4212 raft_consensus.cc:386] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.618371 4212 raft_consensus.cc:748] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.618683 4212 consensus_queue.cc:261] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.620118 4212 ts_tablet_manager.cc:1480] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20261118 02:55:02.620404 4212 tablet_bootstrap.cc:492] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:02.622658 4212 tablet_bootstrap.cc:654] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.627422 4211 tablet_bootstrap.cc:654] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.642204 3598 tablet_service.cc:1500] Processing CreateTablet for tablet e9323addc63245bfacb3854782425347 (DEFAULT_TABLE table=default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.642701 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9323addc63245bfacb3854782425347. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.645967 4210 consensus_queue.cc:1049] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:02.650708 4215 consensus_queue.cc:1049] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:02.667567 4213 tablet_bootstrap.cc:492] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:02.667779 4213 ts_tablet_manager.cc:1449] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.015s
I20261118 02:55:02.668637 4213 raft_consensus.cc:360] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.668875 4213 raft_consensus.cc:386] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.668968 4213 raft_consensus.cc:748] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.669234 4213 consensus_queue.cc:261] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.669760 4213 ts_tablet_manager.cc:1480] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20261118 02:55:02.674525 3735 tablet_service.cc:1500] Processing CreateTablet for tablet e9323addc63245bfacb3854782425347 (DEFAULT_TABLE table=default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.675094 3735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9323addc63245bfacb3854782425347. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.677455 4213 tablet_bootstrap.cc:492] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:02.678975 3882 tablet_service.cc:1500] Processing CreateTablet for tablet e9323addc63245bfacb3854782425347 (DEFAULT_TABLE table=default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:02.679389 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9323addc63245bfacb3854782425347. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:02.680945 4213 tablet_bootstrap.cc:654] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.682927 4230 ts_tablet_manager.cc:1975] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:02.683077 4230 log.cc:1199] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/9a19cb906c974009aa9df99a14a78024
I20261118 02:55:02.683494 4230 ts_tablet_manager.cc:1996] T 9a19cb906c974009aa9df99a14a78024 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:02.684659 3263 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet 9a19cb906c974009aa9df99a14a78024 (table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]) successfully deleted
I20261118 02:55:02.694171 4212 tablet_bootstrap.cc:492] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:02.694375 4212 ts_tablet_manager.cc:1449] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.074s user 0.027s sys 0.023s
I20261118 02:55:02.695160 4212 raft_consensus.cc:360] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.695392 4212 raft_consensus.cc:386] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.695462 4212 raft_consensus.cc:748] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.695758 4212 consensus_queue.cc:261] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.696442 4212 ts_tablet_manager.cc:1480] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261118 02:55:02.697000 4212 tablet_bootstrap.cc:492] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:02.699144 4212 tablet_bootstrap.cc:654] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.718852 4211 tablet_bootstrap.cc:492] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:02.719098 4213 tablet_bootstrap.cc:492] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:02.719250 4213 ts_tablet_manager.cc:1449] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.012s
I20261118 02:55:02.719424 4211 ts_tablet_manager.cc:1449] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.007s
I20261118 02:55:02.720080 4213 raft_consensus.cc:360] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.720315 4213 raft_consensus.cc:386] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.720404 4213 raft_consensus.cc:748] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.720681 4213 consensus_queue.cc:261] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.721235 4213 ts_tablet_manager.cc:1480] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:02.720273 4211 raft_consensus.cc:360] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.722846 4213 tablet_bootstrap.cc:492] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:02.725155 4213 tablet_bootstrap.cc:654] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.733351 4215 raft_consensus.cc:499] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:02.733520 4215 raft_consensus.cc:521] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.734225 4215 leader_election.cc:290] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:02.734817 4211 raft_consensus.cc:386] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.734939 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "65bd6fcaed5b47c3a2478d375ace9636" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:02.735188 3617 raft_consensus.cc:2490] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:02.736142 3838 leader_election.cc:304] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:02.736533 4210 raft_consensus.cc:2826] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:02.736619 4210 raft_consensus.cc:499] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:02.736711 4210 raft_consensus.cc:3096] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.736858 4212 tablet_bootstrap.cc:492] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:02.737006 4212 ts_tablet_manager.cc:1449] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.006s
I20261118 02:55:02.737749 4212 raft_consensus.cc:360] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.737929 4212 raft_consensus.cc:386] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.738003 4212 raft_consensus.cc:748] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.738220 4212 consensus_queue.cc:261] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.738754 4212 ts_tablet_manager.cc:1480] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20261118 02:55:02.739086 4212 tablet_bootstrap.cc:492] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:02.739395 4210 raft_consensus.cc:521] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.740000 4210 leader_election.cc:290] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:02.740880 4212 tablet_bootstrap.cc:654] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.740824 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "65bd6fcaed5b47c3a2478d375ace9636" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:02.741031 3618 raft_consensus.cc:3096] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.742165 4213 tablet_bootstrap.cc:492] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:02.742398 4213 ts_tablet_manager.cc:1449] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
I20261118 02:55:02.743188 4213 raft_consensus.cc:360] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.743439 4213 raft_consensus.cc:386] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.743547 4213 raft_consensus.cc:748] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.743845 3618 raft_consensus.cc:2490] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:02.743884 4213 consensus_queue.cc:261] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.744406 4213 ts_tablet_manager.cc:1480] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20261118 02:55:02.745106 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "65bd6fcaed5b47c3a2478d375ace9636" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:02.745293 3755 raft_consensus.cc:2490] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:02.735004 4211 raft_consensus.cc:748] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.745702 4211 consensus_queue.cc:261] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.746233 4211 ts_tablet_manager.cc:1480] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20261118 02:55:02.746417 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "65bd6fcaed5b47c3a2478d375ace9636" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:02.746644 3752 raft_consensus.cc:3096] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.749362 4212 tablet_bootstrap.cc:492] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:02.749509 4212 ts_tablet_manager.cc:1449] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20261118 02:55:02.749681 3752 raft_consensus.cc:2490] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:02.750262 4212 raft_consensus.cc:360] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.750437 4212 raft_consensus.cc:386] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.750499 4212 raft_consensus.cc:748] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.750845 4211 tablet_bootstrap.cc:492] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:02.750751 4212 consensus_queue.cc:261] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.752421 4212 ts_tablet_manager.cc:1480] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:02.753146 3838 leader_election.cc:304] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:02.753475 4210 raft_consensus.cc:2826] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:02.753621 4210 raft_consensus.cc:705] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:02.753885 4210 consensus_queue.cc:238] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.761008 4211 tablet_bootstrap.cc:654] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.762140 4215 consensus_queue.cc:1049] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:02.762360 4210 consensus_queue.cc:1049] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:02.773140 4211 tablet_bootstrap.cc:492] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:02.773331 4211 ts_tablet_manager.cc:1449] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20261118 02:55:02.774183 4211 raft_consensus.cc:360] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:02.774386 4211 raft_consensus.cc:386] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.774549 4211 raft_consensus.cc:748] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.774864 4211 consensus_queue.cc:261] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:02.775645 4211 ts_tablet_manager.cc:1480] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20261118 02:55:02.775959 4211 tablet_bootstrap.cc:492] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:02.777774 4211 tablet_bootstrap.cc:654] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.795586 4211 tablet_bootstrap.cc:492] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:02.796051 4211 ts_tablet_manager.cc:1449] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20261118 02:55:02.796945 4211 raft_consensus.cc:360] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.797155 4211 raft_consensus.cc:386] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.797224 4211 raft_consensus.cc:748] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.797529 4211 consensus_queue.cc:261] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.798207 4211 ts_tablet_manager.cc:1480] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:02.798553 4211 tablet_bootstrap.cc:492] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:02.800755 4211 tablet_bootstrap.cc:654] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.804750 4211 tablet_bootstrap.cc:492] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:02.804915 4211 ts_tablet_manager.cc:1449] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20261118 02:55:02.805819 4211 raft_consensus.cc:360] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.806030 4211 raft_consensus.cc:386] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.806226 4211 raft_consensus.cc:748] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.806522 4211 consensus_queue.cc:261] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.807209 4211 ts_tablet_manager.cc:1480] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:02.807523 4211 tablet_bootstrap.cc:492] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:02.809410 4211 tablet_bootstrap.cc:654] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:02.839342 4211 tablet_bootstrap.cc:492] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:02.839573 4211 ts_tablet_manager.cc:1449] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20261118 02:55:02.840452 4211 raft_consensus.cc:360] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.840667 4211 raft_consensus.cc:386] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:02.840734 4211 raft_consensus.cc:748] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:02.841044 4211 consensus_queue.cc:261] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.841711 4211 ts_tablet_manager.cc:1480] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20261118 02:55:02.893460 4215 raft_consensus.cc:499] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:02.893675 4215 raft_consensus.cc:521] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.894475 4215 leader_election.cc:290] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:02.895414 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "78189cccd457466bb5d01231038fec24" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:02.895464 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "78189cccd457466bb5d01231038fec24" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:02.895656 3618 raft_consensus.cc:2490] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:02.895713 3752 raft_consensus.cc:2490] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:02.896189 3838 leader_election.cc:304] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:02.896587 4215 raft_consensus.cc:2826] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:02.896709 4215 raft_consensus.cc:499] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:02.896813 4215 raft_consensus.cc:3096] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.899765 4215 raft_consensus.cc:521] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.900928 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "78189cccd457466bb5d01231038fec24" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:02.901141 3618 raft_consensus.cc:3096] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.904482 3618 raft_consensus.cc:2490] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:02.904999 3838 leader_election.cc:304] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:02.905334 4210 raft_consensus.cc:2826] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:02.905485 4210 raft_consensus.cc:705] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:02.906251 4210 consensus_queue.cc:238] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.908751 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "78189cccd457466bb5d01231038fec24" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:02.908952 3752 raft_consensus.cc:3096] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.911922 3752 raft_consensus.cc:2490] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:02.912911 4215 leader_election.cc:290] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:02.922106 4034 raft_consensus.cc:499] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:02.922333 4034 raft_consensus.cc:521] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.923112 4034 leader_election.cc:290] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:02.924456 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "e71c5cd71ebf4fa6bd158d59ed2b191d" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:02.924696 3902 raft_consensus.cc:2490] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:02.925324 3553 leader_election.cc:304] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:02.925747 4034 raft_consensus.cc:2826] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:02.925832 4034 raft_consensus.cc:499] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:02.925917 4034 raft_consensus.cc:3096] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.928529 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "e71c5cd71ebf4fa6bd158d59ed2b191d" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:02.928798 3752 raft_consensus.cc:2490] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:02.930457 4034 raft_consensus.cc:521] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.931615 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "e71c5cd71ebf4fa6bd158d59ed2b191d" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:02.931833 3752 raft_consensus.cc:3096] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.934810 3752 raft_consensus.cc:2490] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:02.935365 3552 leader_election.cc:304] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:02.937553 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "e71c5cd71ebf4fa6bd158d59ed2b191d" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:02.937777 3902 raft_consensus.cc:3096] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:02.941097 3902 raft_consensus.cc:2490] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:02.942000 4034 leader_election.cc:290] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:02.942567 4034 raft_consensus.cc:2826] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:02.942922 4034 raft_consensus.cc:705] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:02.946439 4034 consensus_queue.cc:238] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:02.955217 4215 consensus_queue.cc:1049] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:02.955566 4239 consensus_queue.cc:1049] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:02.961678 4263 consensus_queue.cc:1049] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.024931 4269 consensus_queue.cc:1049] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.079902 4207 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:03.080039 4207 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: loading cluster ID for follower catalog manager: success
I20261118 02:55:03.084290 4207 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: acquiring CA information for follower catalog manager: success
I20261118 02:55:03.087280 4207 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:55:03.121412 4263 raft_consensus.cc:499] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:03.121657 4263 raft_consensus.cc:521] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.122483 4263 leader_election.cc:290] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:03.123921 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "e9323addc63245bfacb3854782425347" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:03.124187 3752 raft_consensus.cc:2490] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:03.124627 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "e9323addc63245bfacb3854782425347" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:03.124868 3902 raft_consensus.cc:2490] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:03.125492 3553 leader_election.cc:304] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:03.126190 4263 raft_consensus.cc:2826] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:03.126302 4263 raft_consensus.cc:499] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:03.126416 4263 raft_consensus.cc:3096] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.130641 4263 raft_consensus.cc:521] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.132220 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "e9323addc63245bfacb3854782425347" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:03.132436 3752 raft_consensus.cc:3096] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.135322 3752 raft_consensus.cc:2490] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:03.135979 3552 leader_election.cc:304] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:03.136355 4270 raft_consensus.cc:2826] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:03.136504 4270 raft_consensus.cc:705] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:03.136807 4270 consensus_queue.cc:238] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.150036 4274 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:55:03.150341 4274 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:03.151278 3902 raft_consensus.cc:3096] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.151767 4274 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:03.151852 4275 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:55:03.152185 4275 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:03.152758 3287 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 4 candidate_status { last_received { term: 3 index: 55 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:55:03.155781 3287 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 55 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:55:03.157490 4034 consensus_queue.cc:1049] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.159027 4034 consensus_queue.cc:1049] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.165925 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "e9323addc63245bfacb3854782425347" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:03.166877 4263 leader_election.cc:290] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:03.172950 4275 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
I20261118 02:55:03.174739 4161 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 4 candidate_status { last_received { term: 3 index: 55 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4" is_pre_election: true
I20261118 02:55:03.175145 4161 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 457eb38cba564ecd9f6f11245172260b in term 3.
I20261118 02:55:03.176805 3333 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d
I20261118 02:55:03.177156 4274 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261118 02:55:03.177296 4274 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:55:03.177381 4274 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 3 FOLLOWER]: Advancing to term 4
I20261118 02:55:03.180444 4274 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:03.181114 4274 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 4 election: Requested vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:03.182090 3287 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 4 candidate_status { last_received { term: 3 index: 55 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d"
I20261118 02:55:03.182473 4161 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 4 candidate_status { last_received { term: 3 index: 55 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4"
I20261118 02:55:03.182662 4161 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 3 FOLLOWER]: Advancing to term 4
I20261118 02:55:03.191579 4161 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 457eb38cba564ecd9f6f11245172260b in term 4.
I20261118 02:55:03.194694 3333 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d
I20261118 02:55:03.195053 4274 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 4 FOLLOWER]: Leader election won for term 4
I20261118 02:55:03.203677 3355 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 4 candidate_status { last_received { term: 3 index: 55 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b" is_pre_election: true
I20261118 02:55:03.204798 4161 raft_consensus.cc:1250] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Rejecting Update request from peer 26c6b20fa203404a88463ea27bdd3c7d for earlier term 3. Current term is 4. Ops: []
I20261118 02:55:03.213188 4283 consensus_queue.cc:1060] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 }, Status: INVALID_TERM, Last received: 3.55, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20261118 02:55:03.218750 4284 raft_consensus.cc:3091] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 3 LEADER]: Stepping down as leader of term 3
I20261118 02:55:03.219136 4284 raft_consensus.cc:748] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 26c6b20fa203404a88463ea27bdd3c7d, State: Running, Role: LEADER
I20261118 02:55:03.219581 4284 consensus_queue.cc:261] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 3.55, Last appended by leader: 55, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:03.220163 4284 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 3 FOLLOWER]: Advancing to term 4
I20261118 02:55:03.221344 4274 raft_consensus.cc:705] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 4 LEADER]: Becoming Leader. State: Replica: 457eb38cba564ecd9f6f11245172260b, State: Running, Role: LEADER
I20261118 02:55:03.221750 4274 consensus_queue.cc:238] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 3.55, Last appended by leader: 55, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:03.223515 4210 raft_consensus.cc:499] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:03.223692 4210 raft_consensus.cc:521] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.224463 4210 leader_election.cc:290] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:03.224998 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2de409a48f5b488484d74008f186d166" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:03.225231 3618 raft_consensus.cc:2490] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:03.225222 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2de409a48f5b488484d74008f186d166" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:03.225425 3755 raft_consensus.cc:2490] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:03.225742 3838 leader_election.cc:304] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:03.226023 4210 raft_consensus.cc:2826] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:03.226126 4210 raft_consensus.cc:499] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:03.226195 4210 raft_consensus.cc:3096] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.227938 4138 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:03.228355 4282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 4 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:03.228598 4282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:03.229053 4275 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261118 02:55:03.229151 4210 raft_consensus.cc:521] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.230233 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2de409a48f5b488484d74008f186d166" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:03.230419 3618 raft_consensus.cc:3096] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.231324 4285 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 02:55:03.233618 3618 raft_consensus.cc:2490] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:03.234088 3838 leader_election.cc:304] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:03.234434 4239 raft_consensus.cc:2826] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:03.234560 4239 raft_consensus.cc:705] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:03.234835 4239 consensus_queue.cc:238] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.237493 4210 leader_election.cc:290] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:03.237653 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2de409a48f5b488484d74008f186d166" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:03.237876 3755 raft_consensus.cc:3096] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.240803 3755 raft_consensus.cc:2490] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:03.242805 4161 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Refusing update from remote peer 457eb38cba564ecd9f6f11245172260b: Log matching property violated. Preceding OpId in replica: term: 3 index: 55. Preceding OpId from leader: term: 4 index: 56. (index mismatch)
I20261118 02:55:03.244297 4282 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Connected to new peer: Peer: permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20261118 02:55:03.247650 4285 catalog_manager.cc:694] Loaded metadata for table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]
I20261118 02:55:03.248421 4285 catalog_manager.cc:694] Loaded metadata for table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]
I20261118 02:55:03.248888 4285 catalog_manager.cc:694] Loaded metadata for table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]
I20261118 02:55:03.249897 4285 catalog_manager.cc:694] Loaded metadata for table default.table_295a3e7e0d1f4e9db7078f6e5a5b9da1 [id=44b390866720455b8d2714eae004383b]
I20261118 02:55:03.250344 4285 catalog_manager.cc:694] Loaded metadata for table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]
I20261118 02:55:03.250787 4285 catalog_manager.cc:694] Loaded metadata for table default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da]
I20261118 02:55:03.251181 4285 catalog_manager.cc:694] Loaded metadata for table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]
I20261118 02:55:03.251591 4285 catalog_manager.cc:694] Loaded metadata for table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]
I20261118 02:55:03.254302 3287 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 4 FOLLOWER]: Refusing update from remote peer 457eb38cba564ecd9f6f11245172260b: Log matching property violated. Preceding OpId in replica: term: 3 index: 55. Preceding OpId from leader: term: 4 index: 56. (index mismatch)
I20261118 02:55:03.255494 4288 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20261118 02:55:03.257939 4210 consensus_queue.cc:1049] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.258652 4284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 4 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:03.258842 4284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:03.259121 4275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 4 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:03.259346 4275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:03.260438 4210 consensus_queue.cc:1049] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.283639 4285 catalog_manager.cc:694] Loaded metadata for table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]
I20261118 02:55:03.284159 4285 catalog_manager.cc:694] Loaded metadata for table default.table_df2a47bb353f4cdcb4c16399a99cb722 [id=8a5ea4b95bee4db7af9ba875250d564b]
I20261118 02:55:03.284605 4285 catalog_manager.cc:694] Loaded metadata for table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]
I20261118 02:55:03.285006 4285 catalog_manager.cc:694] Loaded metadata for table default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7]
I20261118 02:55:03.285444 4285 catalog_manager.cc:694] Loaded metadata for table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]
I20261118 02:55:03.285876 4285 catalog_manager.cc:694] Loaded metadata for table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]
I20261118 02:55:03.295509 4282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:03.296013 4274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:03.296324 4274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:03.296545 4282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:03.308308 4285 tablet_loader.cc:96] loaded metadata for tablet 1d347c0ae53241708ec623562f4740d4 (table default.table_df2a47bb353f4cdcb4c16399a99cb722 [id=8a5ea4b95bee4db7af9ba875250d564b])
I20261118 02:55:03.310734 4275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:03.310953 4275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:03.311246 4285 tablet_loader.cc:96] loaded metadata for tablet 2de409a48f5b488484d74008f186d166 (table default.table_0006608f28004098bb49ed40308f98fe [id=a2ece7af36934cdf80ad8458024adec7])
I20261118 02:55:03.311656 4285 tablet_loader.cc:96] loaded metadata for tablet 489beea71ae74236addee59c153cfee7 (table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e])
I20261118 02:55:03.312568 4285 tablet_loader.cc:96] loaded metadata for tablet 55f81e7b62e54ec18c7c13a200edb68f (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db])
I20261118 02:55:03.315150 4285 tablet_loader.cc:96] loaded metadata for tablet 5f05a8e5b21945a5bd3df86a5467a615 (table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf])
I20261118 02:55:03.315614 4285 tablet_loader.cc:96] loaded metadata for tablet 65bd6fcaed5b47c3a2478d375ace9636 (table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73])
I20261118 02:55:03.315896 4284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:03.316133 4284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:03.316834 4285 tablet_loader.cc:96] loaded metadata for tablet 78189cccd457466bb5d01231038fec24 (table default.table_f777ce1b15a4431fa208932de26f41fb [id=4a09f210503b499f8e842f6daafb33da])
I20261118 02:55:03.322645 4285 tablet_loader.cc:96] loaded metadata for tablet 8f853e6b923444efa79cfbab2e31540e (table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec])
I20261118 02:55:03.323632 4285 tablet_loader.cc:96] loaded metadata for tablet a013d180f2cd430ab1ba30b39fd27591 (table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d])
I20261118 02:55:03.324040 4285 tablet_loader.cc:96] loaded metadata for tablet a95a3f25901e468ab6e6cd02c2cd035f (table default.table_295a3e7e0d1f4e9db7078f6e5a5b9da1 [id=44b390866720455b8d2714eae004383b])
I20261118 02:55:03.324433 4285 tablet_loader.cc:96] loaded metadata for tablet c92c29c30ab6411dabe6f02331d29b7a (table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d])
I20261118 02:55:03.325109 4285 tablet_loader.cc:96] loaded metadata for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6])
I20261118 02:55:03.325471 4285 tablet_loader.cc:96] loaded metadata for tablet e9323addc63245bfacb3854782425347 (table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e])
I20261118 02:55:03.325857 4285 tablet_loader.cc:96] loaded metadata for tablet ffe8d4c111d14054b76f9c7724072a94 (table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66])
I20261118 02:55:03.326162 4285 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 02:55:03.327667 4285 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:03.327769 4285 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:55:03.329511 4285 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:55:03.341220 4239 raft_consensus.cc:499] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:03.341445 4239 raft_consensus.cc:521] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.342236 4239 leader_election.cc:290] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:03.343024 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "489beea71ae74236addee59c153cfee7" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:03.343281 3617 raft_consensus.cc:2490] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:03.343673 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "489beea71ae74236addee59c153cfee7" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:03.343817 3838 leader_election.cc:304] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:03.343917 3755 raft_consensus.cc:2490] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:03.345257 4285 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Loaded TSK: 0
I20261118 02:55:03.345374 4239 raft_consensus.cc:2826] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:03.345466 4239 raft_consensus.cc:499] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:03.345558 4239 raft_consensus.cc:3096] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.345836 4285 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 02:55:03.347131 4285 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:55:03.348455 4285 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: 11
I20261118 02:55:03.348922 4239 raft_consensus.cc:521] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.350570 3617 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "489beea71ae74236addee59c153cfee7" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:03.350839 3617 raft_consensus.cc:3096] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.351061 3755 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "489beea71ae74236addee59c153cfee7" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:03.351289 3755 raft_consensus.cc:3096] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.352309 4239 leader_election.cc:290] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:03.354066 3755 raft_consensus.cc:2490] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:03.354327 3617 raft_consensus.cc:2490] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:03.354602 3836 leader_election.cc:304] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:03.355382 4239 raft_consensus.cc:2826] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:03.356189 4239 raft_consensus.cc:705] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:03.356560 4239 consensus_queue.cc:238] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.377735 3346 catalog_manager.cc:5840] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: HEALTHY } } }
I20261118 02:55:03.391562 4210 consensus_queue.cc:1049] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.399912 4239 consensus_queue.cc:1049] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 02:55:03.409021 3482 hms_client.cc:355] Time spent drop HMS table: real 1.047s user 0.000s sys 0.000s
I20261118 02:55:03.409365 3269 catalog_manager.cc:2772] deleted HMS catalog entry for table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]
I20261118 02:55:03.429870 4214 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:03.430089 4214 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:03.430902 4214 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:03.431723 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a013d180f2cd430ab1ba30b39fd27591" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:03.431933 3618 raft_consensus.cc:2490] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:03.432497 3691 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
W20261118 02:55:03.432767 3486 catalog_manager.cc:3980] Illegal state: an error occurred while updating the sys-catalog: replica 26c6b20fa203404a88463ea27bdd3c7d is not leader of this config: current role FOLLOWER
I20261118 02:55:03.432950 4214 raft_consensus.cc:2826] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:03.433051 4214 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20261118 02:55:03.433043 3486 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 12 ALTER_TABLE default.table_30f082fbe850451b80d98df75ed1081a: Illegal state: an error occurred while updating the sys-catalog: replica 26c6b20fa203404a88463ea27bdd3c7d is not leader of this config: current role FOLLOWER
I20261118 02:55:03.433125 4214 raft_consensus.cc:3096] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.433245 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a013d180f2cd430ab1ba30b39fd27591" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:03.433480 3901 raft_consensus.cc:2490] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:03.436354 4214 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:03.437338 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a013d180f2cd430ab1ba30b39fd27591" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:03.437531 3901 raft_consensus.cc:3096] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.437681 3618 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a013d180f2cd430ab1ba30b39fd27591" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:03.437853 3618 raft_consensus.cc:3096] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.440456 3901 raft_consensus.cc:2490] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:03.440697 3618 raft_consensus.cc:2490] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:03.441155 3691 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:03.442617 4214 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:03.442819 4214 raft_consensus.cc:2826] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:03.442945 4214 raft_consensus.cc:705] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:03.443228 4214 consensus_queue.cc:238] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
W20261118 02:55:03.446483 3486 hms_notification_log_listener.cc:143] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events: an error occurred while updating the sys-catalog: replica 26c6b20fa203404a88463ea27bdd3c7d is not leader of this config: current role FOLLOWER
I20261118 02:55:03.446903 3277 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 26) took 2767 ms. Trace:
I20261118 02:55:03.447062 3277 rpcz_store.cc:276] 1118 02:55:00.679452 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:01.871270 (+1191818us) service_pool.cc:224] Handling call
1118 02:55:03.446837 (+1575567us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261118 02:55:03.447537 3275 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:52040 (request call id 23) took 2967 ms. Trace:
W20261118 02:55:03.447603 3270 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_51d36ff687ca47b5995e5693d7b8931b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261118 02:55:03.448141 3275 rpcz_store.cc:276] 1118 02:55:00.479705 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:00.888048 (+408343us) service_pool.cc:224] Handling call
1118 02:55:01.677032 (+788984us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:03.447026 (+1769994us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:03.447241 (+ 215us) catalog_manager.cc:3766] Apply alter schema
1118 02:55:03.447490 (+ 249us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":9455,"Hive Metastore.run_cpu_time_us":655,"Hive Metastore.run_wall_time_us":194702,"spinlock_wait_cycles":2816}
W20261118 02:55:03.448364 3277 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.2.229.66:52737 (request call id 17) took 900 ms (client timeout 1000 ms). Trace:
W20261118 02:55:03.448493 3277 rpcz_store.cc:269] 1118 02:55:02.547455 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.447366 (+899911us) service_pool.cc:224] Handling call
1118 02:55:03.448321 (+ 955us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20261118 02:55:03.449167 3275 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.2.229.65:59011 (request call id 21) took 900 ms (client timeout 1000 ms). Trace:
W20261118 02:55:03.449273 3275 rpcz_store.cc:269] 1118 02:55:02.548572 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.448446 (+899874us) service_pool.cc:224] Handling call
1118 02:55:03.449125 (+ 679us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20261118 02:55:03.449621 3277 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.2.229.67:38945 (request call id 17) took 801 ms (client timeout 1000 ms). Trace:
I20261118 02:55:03.449750 3275 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 36) took 2514 ms. Trace:
I20261118 02:55:03.449872 3275 rpcz_store.cc:276] 1118 02:55:00.934962 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.449418 (+2514456us) service_pool.cc:224] Handling call
1118 02:55:03.449698 (+ 280us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20261118 02:55:03.451026 3956 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.229.126:38025)
I20261118 02:55:03.451393 3274 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:52040 (request call id 12) took 3235 ms. Trace:
I20261118 02:55:03.451489 3274 rpcz_store.cc:276] 1118 02:55:00.216389 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:00.216823 (+ 434us) service_pool.cc:224] Handling call
1118 02:55:01.677233 (+1460410us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:03.450814 (+1773581us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:03.451046 (+ 232us) catalog_manager.cc:3766] Apply alter schema
1118 02:55:03.451357 (+ 311us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":204066,"Hive Metastore.run_cpu_time_us":790,"Hive Metastore.run_wall_time_us":83922}
I20261118 02:55:03.452023 3274 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 38) took 2383 ms. Trace:
I20261118 02:55:03.454147 3274 rpcz_store.cc:276] 1118 02:55:01.068231 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.451732 (+2383501us) service_pool.cc:224] Handling call
1118 02:55:03.451993 (+ 261us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20261118 02:55:03.449752 3277 rpcz_store.cc:269] 1118 02:55:02.647718 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.448750 (+801032us) service_pool.cc:224] Handling call
1118 02:55:03.449586 (+ 836us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261118 02:55:03.455120 3272 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:52040 (request call id 17) took 3100 ms. Trace:
I20261118 02:55:03.455209 3272 rpcz_store.cc:276] 1118 02:55:00.352343 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:00.568189 (+215846us) service_pool.cc:224] Handling call
1118 02:55:01.677804 (+1109615us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:03.452204 (+1774400us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:03.452416 (+ 212us) catalog_manager.cc:3766] Apply alter schema
1118 02:55:03.452656 (+ 240us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":287494,"Hive Metastore.run_cpu_time_us":777,"Hive Metastore.run_wall_time_us":79853}
I20261118 02:55:03.455560 3269 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:52040 (request call id 25) took 2816 ms. Trace:
I20261118 02:55:03.455972 3269 rpcz_store.cc:276] 1118 02:55:00.637027 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:01.076161 (+439134us) service_pool.cc:224] Handling call
1118 02:55:01.751224 (+675063us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:03.453248 (+1702024us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":610526,"Hive Metastore.run_cpu_time_us":387,"Hive Metastore.run_wall_time_us":1047193}
I20261118 02:55:03.456377 3344 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48092:
name: "default.Table_f0ae490a0df64e1097e144c7a370df61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:03.457382 3347 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48092:
name: "default.Table_571d1360811c42dc815efb439b2b52e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:03.460268 3275 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:52040 (request call id 37) took 2433 ms. Trace:
I20261118 02:55:03.460486 3276 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:52040 (request call id 13) took 3211 ms. Trace:
I20261118 02:55:03.460832 3276 rpcz_store.cc:276] 1118 02:55:00.244156 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:00.244212 (+ 56us) service_pool.cc:224] Handling call
1118 02:55:01.677032 (+1432820us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:03.452890 (+1775858us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:03.453077 (+ 187us) catalog_manager.cc:3766] Apply alter schema
1118 02:55:03.455873 (+ 2796us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":420452,"Hive Metastore.run_cpu_time_us":727,"Hive Metastore.run_wall_time_us":140665,"mutex_wait_us":1539}
I20261118 02:55:03.463210 3272 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:52040 (request call id 41) took 1505 ms. Trace:
I20261118 02:55:03.463317 3272 rpcz_store.cc:276] 1118 02:55:01.957099 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.455422 (+1498323us) service_pool.cc:224] Handling call
1118 02:55:03.461787 (+ 6365us) spinlock_profiling.cc:245] Waited 5.75 ms on lock 0x616000043354. stack: 00007f9c4434a3a6 00007f9c4434a160 00007f9c4333b34d 00007f9c5a90e704 00007f9c5a90e6cd 00007f9c5386082d 00007f9c53860923 00007f9c503ca221 00007f9c582bf50d 00007f9c58450b06 00007f9c4ed5aea6 00007f9c4ed5abd5 00007f9c4b4928de 00007f9c4b4917be 00007f9c4b4979f8 00007f9c4b49835f
1118 02:55:03.462019 (+ 232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13217152}
I20261118 02:55:03.463842 4300 consensus_queue.cc:1049] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.464421 4214 consensus_queue.cc:1049] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.465302 3272 master_service.cc:965] ReplaceTablet: received request to replace tablet a95a3f25901e468ab6e6cd02c2cd035f from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:03.466070 3272 master_service.cc:965] ReplaceTablet: received request to replace tablet a95a3f25901e468ab6e6cd02c2cd035f from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:03.466806 3272 master_service.cc:965] ReplaceTablet: received request to replace tablet 55f81e7b62e54ec18c7c13a200edb68f from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:03.467231 3272 master_service.cc:965] ReplaceTablet: received request to replace tablet 2de409a48f5b488484d74008f186d166 from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:03.466634 3275 rpcz_store.cc:276] 1118 02:55:01.027202 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.450042 (+2422840us) service_pool.cc:224] Handling call
1118 02:55:03.460228 (+ 10186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261118 02:55:03.468441 3269 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 42) took 1155 ms. Trace:
I20261118 02:55:03.468607 3272 master_service.cc:965] ReplaceTablet: received request to replace tablet e9323addc63245bfacb3854782425347 from {username='slave'} at 127.0.0.1:52040
I20261118 02:55:03.468904 3274 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 39) took 2355 ms. Trace:
I20261118 02:55:03.469182 3269 rpcz_store.cc:276] 1118 02:55:02.309431 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.456276 (+1146845us) service_pool.cc:224] Handling call
1118 02:55:03.462319 (+ 6043us) spinlock_profiling.cc:245] Waited 5.81 ms on lock 0x616000043354. stack: 00007f9c4434a3a6 00007f9c4434a160 00007f9c4333b34d 00007f9c5a90e704 00007f9c5a90e6cd 00007f9c5386082d 00007f9c53860923 00007f9c503ca221 00007f9c582bf50d 00007f9c5844e873 00007f9c4ed57526 00007f9c4ed57255 00007f9c4b4928de 00007f9c4b4917be 00007f9c4b4979f8 00007f9c4b49835f
1118 02:55:03.465095 (+ 2776us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13373184}
I20261118 02:55:03.469892 3277 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 40) took 2237 ms. Trace:
I20261118 02:55:03.471904 3274 rpcz_store.cc:276] 1118 02:55:01.107611 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.454331 (+2346720us) service_pool.cc:224] Handling call
1118 02:55:03.462681 (+ 8350us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261118 02:55:03.474150 3277 rpcz_store.cc:276] 1118 02:55:01.227610 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.454741 (+2227131us) service_pool.cc:224] Handling call
1118 02:55:03.462860 (+ 8119us) spinlock_profiling.cc:245] Waited 7.69 ms on lock 0x616000043354. stack: 00007f9c4434a3a6 00007f9c4434a160 00007f9c4333b34d 00007f9c5a90e704 00007f9c5a90e6cd 00007f9c5386082d 00007f9c53860923 00007f9c503ca221 00007f9c582bf50d 00007f9c5844e873 00007f9c4ed57526 00007f9c4ed57255 00007f9c4b4928de 00007f9c4b4917be 00007f9c4b4979f8 00007f9c4b49835f
1118 02:55:03.465237 (+ 2377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17687808}
I20261118 02:55:03.476303 3275 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:52050 (request call id 0) took 1338 ms. Trace:
I20261118 02:55:03.476471 3275 rpcz_store.cc:276] 1118 02:55:02.138073 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.471169 (+1333096us) service_pool.cc:224] Handling call
1118 02:55:03.476264 (+ 5095us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261118 02:55:03.481215 3949 heartbeater.cc:500] Master 127.2.229.125:38339 was elected leader, sending a full tablet report...
I20261118 02:55:03.482980 3802 heartbeater.cc:500] Master 127.2.229.125:38339 was elected leader, sending a full tablet report...
I20261118 02:55:03.487481 3345 catalog_manager.cc:5840] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66). New cstate: current_term: 1 leader_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20261118 02:55:03.492015 3345 catalog_manager.cc:5840] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } }
I20261118 02:55:03.494798 3345 catalog_manager.cc:5840] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } }
I20261118 02:55:03.495751 3345 catalog_manager.cc:5940] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377) does not have the latest schema for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]). Expected version 1 got 0
I20261118 02:55:03.496626 3345 catalog_manager.cc:5840] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } }
I20261118 02:55:03.497951 3345 catalog_manager.cc:5840] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } }
I20261118 02:55:03.498889 3345 catalog_manager.cc:5840] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } }
I20261118 02:55:03.499925 3345 catalog_manager.cc:5840] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } }
I20261118 02:55:03.513034 3665 heartbeater.cc:500] Master 127.2.229.125:38339 was elected leader, sending a full tablet report...
I20261118 02:55:03.517779 3346 catalog_manager.cc:5940] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237) does not have the latest schema for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]). Expected version 1 got 0
I20261118 02:55:03.518090 3346 catalog_manager.cc:5940] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237) does not have the latest schema for tablet a95a3f25901e468ab6e6cd02c2cd035f (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]). Expected version 1 got 0
I20261118 02:55:03.521122 3343 catalog_manager.cc:5940] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705) does not have the latest schema for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]). Expected version 1 got 0
I20261118 02:55:03.521386 3343 catalog_manager.cc:5940] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705) does not have the latest schema for tablet a95a3f25901e468ab6e6cd02c2cd035f (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]). Expected version 1 got 0
I20261118 02:55:03.535491 3343 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48092:
name: "default.Table_30b09a2798a54b01be7618b8a28eafaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261118 02:55:03.545367 3956 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.229.125:38339)
I20261118 02:55:03.545638 3342 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48092:
table { table_name: "default.Table_8ae376d4d4d64414b5c08dc3c2e58bc4" } new_table_name: "default.Table_5091d82298ca44bea5335d4989eb02d8" modify_external_catalogs: true
I20261118 02:55:03.552434 3341 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48092:
name: "default.Table_61b6eaac749b4be0b0ddfba6ff9aa706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:03.560573 3340 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48092:
name: "default.Table_6e09d0e89581458183ed524cc1404849"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:03.564914 3339 master_service.cc:965] ReplaceTablet: received request to replace tablet a95a3f25901e468ab6e6cd02c2cd035f from {username='slave'} at 127.0.0.1:48092
W20261118 02:55:03.564642 3956 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.229.125:38339)
W20261118 02:55:03.579213 3691 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.229.65:33705, user_credentials={real_user=slave}} blocked reactor thread for 117228us
I20261118 02:55:03.579779 3345 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48092:
name: "default.Table_38304cbb980440c2a8b8a6c13f49949c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:03.582274 3339 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet a95a3f25901e468ab6e6cd02c2cd035f
I20261118 02:55:03.583256 3339 catalog_manager.cc:6936] ReplaceTablet: tablet a95a3f25901e468ab6e6cd02c2cd035f deleted and replaced by tablet 72f427c72a554ac1a0a2f2c3a78865da
W20261118 02:55:03.589473 3956 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.229.125:38339)
I20261118 02:55:03.591048 3339 master_service.cc:965] ReplaceTablet: received request to replace tablet a95a3f25901e468ab6e6cd02c2cd035f from {username='slave'} at 127.0.0.1:48092
I20261118 02:55:03.591394 3338 master_service.cc:965] ReplaceTablet: received request to replace tablet 2de409a48f5b488484d74008f186d166 from {username='slave'} at 127.0.0.1:48092
I20261118 02:55:03.592765 3339 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48092:
table { table_name: "default.Table_28c62f411da74545aa42dad7325bc064" } modify_external_catalogs: true
I20261118 02:55:03.593070 3339 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48092:
table { table_name: "default.Table_28c62f411da74545aa42dad7325bc064" } modify_external_catalogs: true
I20261118 02:55:03.593319 3346 master_service.cc:965] ReplaceTablet: received request to replace tablet 55f81e7b62e54ec18c7c13a200edb68f from {username='slave'} at 127.0.0.1:48092
I20261118 02:55:03.621941 3880 tablet_service.cc:1547] Processing DeleteTablet for tablet a95a3f25901e468ab6e6cd02c2cd035f with delete_type TABLET_DATA_DELETED (replaced by tablet 72f427c72a554ac1a0a2f2c3a78865da) from {username='slave'} at 127.0.0.1:50250
I20261118 02:55:03.622697 3879 tablet_service.cc:1500] Processing CreateTablet for tablet 72f427c72a554ac1a0a2f2c3a78865da (DEFAULT_TABLE table=default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:03.623270 3879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f427c72a554ac1a0a2f2c3a78865da. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:03.624219 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet a95a3f25901e468ab6e6cd02c2cd035f with delete_type TABLET_DATA_DELETED (replaced by tablet 72f427c72a554ac1a0a2f2c3a78865da) from {username='slave'} at 127.0.0.1:53002
I20261118 02:55:03.624902 3595 tablet_service.cc:1500] Processing CreateTablet for tablet 72f427c72a554ac1a0a2f2c3a78865da (DEFAULT_TABLE table=default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:03.625432 3595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f427c72a554ac1a0a2f2c3a78865da. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:03.666306 4321 tablet.cc:1759] T e71c5cd71ebf4fa6bd158d59ed2b191d P 1fdf7d7ece324fefb3a0c376add5b7e8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.667302 4322 tablet.cc:1759] T e71c5cd71ebf4fa6bd158d59ed2b191d P 99a39b1d9d814deeb3a8a644e9b392ef: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.673839 3338 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet 2de409a48f5b488484d74008f186d166
I20261118 02:55:03.674403 3338 catalog_manager.cc:6936] ReplaceTablet: tablet 2de409a48f5b488484d74008f186d166 deleted and replaced by tablet 01778879aaa44ea9855fae67cfb3952d
I20261118 02:55:03.675971 3346 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet 55f81e7b62e54ec18c7c13a200edb68f
I20261118 02:55:03.676450 3346 catalog_manager.cc:6936] ReplaceTablet: tablet 55f81e7b62e54ec18c7c13a200edb68f deleted and replaced by tablet 2b01d7e596bd4790b8ba5dc88d8e12cc
I20261118 02:55:03.678709 3346 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48092:
table { table_name: "default.Table_a14cb6414555446286bf35bc5c6fb910" } new_table_name: "default.Table_324a2600fef947748ef03e27f7cb7e09" modify_external_catalogs: true
I20261118 02:55:03.681584 3882 tablet_service.cc:1547] Processing DeleteTablet for tablet 2de409a48f5b488484d74008f186d166 with delete_type TABLET_DATA_DELETED (replaced by tablet 01778879aaa44ea9855fae67cfb3952d) from {username='slave'} at 127.0.0.1:50250
I20261118 02:55:03.682271 3882 tablet_service.cc:1547] Processing DeleteTablet for tablet 55f81e7b62e54ec18c7c13a200edb68f with delete_type TABLET_DATA_DELETED (replaced by tablet 2b01d7e596bd4790b8ba5dc88d8e12cc) from {username='slave'} at 127.0.0.1:50250
I20261118 02:55:03.683172 3338 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48092:
name: "default.Table_d4fec0aaed15433ea373740872db53d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:03.683820 3527 catalog_manager.cc:2813] Deleting table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e] in response to Hive Metastore notification log event 20
I20261118 02:55:03.695082 3882 tablet_service.cc:1500] Processing CreateTablet for tablet 01778879aaa44ea9855fae67cfb3952d (DEFAULT_TABLE table=default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:03.695530 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01778879aaa44ea9855fae67cfb3952d. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:03.704857 3527 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet e9323addc63245bfacb3854782425347
W20261118 02:55:03.706442 3344 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f0ae490a0df64e1097e144c7a370df61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:03.708050 3347 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_571d1360811c42dc815efb439b2b52e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261118 02:55:03.708411 3878 tablet_service.cc:1547] Processing DeleteTablet for tablet e9323addc63245bfacb3854782425347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-18 02:55:03 UTC) from {username='slave'} at 127.0.0.1:50250
I20261118 02:55:03.709728 3882 tablet_service.cc:1500] Processing CreateTablet for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc (DEFAULT_TABLE table=default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:03.710244 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:03.731057 4321 tablet.cc:1759] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.745338 4333 tablet.cc:1759] T 78189cccd457466bb5d01231038fec24 P 3f1973986ec1466eab5bec01dd81654f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.745716 3596 tablet_service.cc:1547] Processing DeleteTablet for tablet 2de409a48f5b488484d74008f186d166 with delete_type TABLET_DATA_DELETED (replaced by tablet 01778879aaa44ea9855fae67cfb3952d) from {username='slave'} at 127.0.0.1:53002
I20261118 02:55:03.755059 4336 tablet.cc:1759] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.759779 4337 tablet.cc:1759] T 78189cccd457466bb5d01231038fec24 P 99a39b1d9d814deeb3a8a644e9b392ef: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.761742 4322 tablet.cc:1759] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.765602 3593 tablet_service.cc:1547] Processing DeleteTablet for tablet 55f81e7b62e54ec18c7c13a200edb68f with delete_type TABLET_DATA_DELETED (replaced by tablet 2b01d7e596bd4790b8ba5dc88d8e12cc) from {username='slave'} at 127.0.0.1:53002
I20261118 02:55:03.766404 3593 tablet_service.cc:1500] Processing CreateTablet for tablet 01778879aaa44ea9855fae67cfb3952d (DEFAULT_TABLE table=default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:03.767023 3593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01778879aaa44ea9855fae67cfb3952d. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:03.772742 3596 tablet_service.cc:1500] Processing CreateTablet for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc (DEFAULT_TABLE table=default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:03.773170 3596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:03.774120 3592 tablet_service.cc:1547] Processing DeleteTablet for tablet e9323addc63245bfacb3854782425347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-18 02:55:03 UTC) from {username='slave'} at 127.0.0.1:53002
I20261118 02:55:03.774791 3735 tablet_service.cc:1547] Processing DeleteTablet for tablet a95a3f25901e468ab6e6cd02c2cd035f with delete_type TABLET_DATA_DELETED (replaced by tablet 72f427c72a554ac1a0a2f2c3a78865da) from {username='slave'} at 127.0.0.1:57892
I20261118 02:55:03.779318 4338 tablet_replica.cc:333] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:03.779599 3735 tablet_service.cc:1500] Processing CreateTablet for tablet 72f427c72a554ac1a0a2f2c3a78865da (DEFAULT_TABLE table=default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:03.780071 3735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f427c72a554ac1a0a2f2c3a78865da. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:03.786352 4330 tablet.cc:1759] T 78189cccd457466bb5d01231038fec24 P 1fdf7d7ece324fefb3a0c376add5b7e8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.792393 4330 tablet.cc:1759] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.799094 4333 tablet.cc:1759] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.799515 4335 tablet.cc:1759] T e71c5cd71ebf4fa6bd158d59ed2b191d P 3f1973986ec1466eab5bec01dd81654f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.800868 4317 tablet_replica.cc:333] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:03.801453 4317 raft_consensus.cc:2261] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:03.801795 4317 raft_consensus.cc:2290] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:03.802723 4317 ts_tablet_manager.cc:1962] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:03.804030 4323 tablet_bootstrap.cc:492] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:03.804560 3734 tablet_service.cc:1547] Processing DeleteTablet for tablet 2de409a48f5b488484d74008f186d166 with delete_type TABLET_DATA_DELETED (replaced by tablet 01778879aaa44ea9855fae67cfb3952d) from {username='slave'} at 127.0.0.1:57892
I20261118 02:55:03.804627 3733 tablet_service.cc:1547] Processing DeleteTablet for tablet 55f81e7b62e54ec18c7c13a200edb68f with delete_type TABLET_DATA_DELETED (replaced by tablet 2b01d7e596bd4790b8ba5dc88d8e12cc) from {username='slave'} at 127.0.0.1:57892
I20261118 02:55:03.805130 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 01778879aaa44ea9855fae67cfb3952d (DEFAULT_TABLE table=default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:03.805495 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01778879aaa44ea9855fae67cfb3952d. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:03.806635 4323 tablet_bootstrap.cc:654] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:03.808066 4328 tablet_bootstrap.cc:492] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:03.810057 4328 tablet_bootstrap.cc:654] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:03.810580 4341 tablet_bootstrap.cc:492] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:03.811090 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc (DEFAULT_TABLE table=default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:03.811463 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:03.814710 4341 tablet_bootstrap.cc:654] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:03.820399 4338 raft_consensus.cc:2261] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:03.820902 4338 raft_consensus.cc:2290] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:03.822979 4338 ts_tablet_manager.cc:1962] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:03.824615 4316 tablet_replica.cc:333] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:03.824888 4316 raft_consensus.cc:2261] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:03.825148 4316 pending_rounds.cc:70] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: Trying to abort 1 pending ops.
I20261118 02:55:03.825354 4316 pending_rounds.cc:77] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7352199182850248704 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "a95a3f25901e468ab6e6cd02c2cd035f" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.table_30f082fbe850451b80d98df75ed1081a" dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" new_extra_config { } }
I20261118 02:55:03.825989 4316 raft_consensus.cc:2290] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:03.830727 4338 ts_tablet_manager.cc:1975] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:03.830931 4338 log.cc:1199] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/a95a3f25901e468ab6e6cd02c2cd035f
I20261118 02:55:03.831468 4338 ts_tablet_manager.cc:1996] T a95a3f25901e468ab6e6cd02c2cd035f P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:03.832007 4338 tablet_replica.cc:333] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:03.832305 4338 raft_consensus.cc:2261] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:03.832566 4338 raft_consensus.cc:2290] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:03.833137 3332 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet a95a3f25901e468ab6e6cd02c2cd035f (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]) successfully deleted
I20261118 02:55:03.834319 4338 ts_tablet_manager.cc:1962] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:03.835621 4321 tablet.cc:1759] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.836117 4316 ts_tablet_manager.cc:1962] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:03.840148 3735 tablet_service.cc:1547] Processing DeleteTablet for tablet e9323addc63245bfacb3854782425347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-18 02:55:03 UTC) from {username='slave'} at 127.0.0.1:57892
I20261118 02:55:03.841574 4338 ts_tablet_manager.cc:1975] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:03.841698 4338 log.cc:1199] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/2de409a48f5b488484d74008f186d166
I20261118 02:55:03.842047 4338 ts_tablet_manager.cc:1996] T 2de409a48f5b488484d74008f186d166 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:03.842489 4338 tablet_replica.cc:333] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:03.842674 4338 raft_consensus.cc:2261] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:03.842864 4338 raft_consensus.cc:2290] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:03.843048 3332 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet 2de409a48f5b488484d74008f186d166 (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]) successfully deleted
I20261118 02:55:03.843485 4335 tablet.cc:1759] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.844619 4328 tablet_bootstrap.cc:492] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:03.844769 4328 ts_tablet_manager.cc:1449] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.003s
I20261118 02:55:03.845530 4328 raft_consensus.cc:360] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:03.845713 4328 raft_consensus.cc:386] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:03.845803 4328 raft_consensus.cc:748] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:03.846498 4328 consensus_queue.cc:261] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:03.847155 4328 ts_tablet_manager.cc:1480] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:03.848531 4323 tablet_bootstrap.cc:492] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:03.848742 4323 ts_tablet_manager.cc:1449] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.001s
I20261118 02:55:03.849772 4323 raft_consensus.cc:360] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:03.850015 4323 raft_consensus.cc:386] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:03.850136 4323 raft_consensus.cc:748] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:03.850394 4323 consensus_queue.cc:261] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:03.851122 4323 ts_tablet_manager.cc:1480] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261118 02:55:03.851469 4323 tablet_bootstrap.cc:492] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:03.853374 4323 tablet_bootstrap.cc:654] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:03.857371 4341 tablet_bootstrap.cc:492] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:03.857643 4341 ts_tablet_manager.cc:1449] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.011s
I20261118 02:55:03.858532 4341 raft_consensus.cc:360] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.858798 4341 raft_consensus.cc:386] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:03.858952 4341 raft_consensus.cc:748] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:03.859251 4341 consensus_queue.cc:261] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.860420 4332 tablet.cc:1759] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:03.861956 4341 ts_tablet_manager.cc:1480] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20261118 02:55:03.863229 4341 tablet_bootstrap.cc:492] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:03.867128 4341 tablet_bootstrap.cc:654] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:03.869217 4338 ts_tablet_manager.cc:1962] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:03.875099 4323 tablet_bootstrap.cc:492] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:03.875352 4323 ts_tablet_manager.cc:1449] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20261118 02:55:03.876965 4323 raft_consensus.cc:360] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.877614 4323 raft_consensus.cc:386] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:03.877737 4323 raft_consensus.cc:748] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:03.880014 4323 consensus_queue.cc:261] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.880988 4323 ts_tablet_manager.cc:1480] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 02:55:03.881300 4323 tablet_bootstrap.cc:492] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:03.881603 4325 raft_consensus.cc:499] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:03.877444 4317 ts_tablet_manager.cc:1975] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:03.881888 4317 log.cc:1199] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/a95a3f25901e468ab6e6cd02c2cd035f
I20261118 02:55:03.881789 4325 raft_consensus.cc:521] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:03.882475 4317 ts_tablet_manager.cc:1996] T a95a3f25901e468ab6e6cd02c2cd035f P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:03.882546 4325 leader_election.cc:290] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:03.883306 4328 tablet_bootstrap.cc:492] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:03.884974 4323 tablet_bootstrap.cc:654] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:03.885033 4328 tablet_bootstrap.cc:654] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:03.887609 3753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "72f427c72a554ac1a0a2f2c3a78865da" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
W20261118 02:55:03.888275 3836 leader_election.cc:343] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Illegal state: must be running to vote when last-logged opid is not known
I20261118 02:55:03.889134 4338 ts_tablet_manager.cc:1975] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:03.889251 4338 log.cc:1199] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/55f81e7b62e54ec18c7c13a200edb68f
I20261118 02:55:03.889629 4338 ts_tablet_manager.cc:1996] T 55f81e7b62e54ec18c7c13a200edb68f P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:03.890619 3332 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet 55f81e7b62e54ec18c7c13a200edb68f (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]) successfully deleted
I20261118 02:55:03.891647 4341 tablet_bootstrap.cc:492] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:03.892009 4341 ts_tablet_manager.cc:1449] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.005s
I20261118 02:55:03.892843 4341 raft_consensus.cc:360] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:03.893030 4341 raft_consensus.cc:386] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:03.893100 4341 raft_consensus.cc:748] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:03.893326 4341 consensus_queue.cc:261] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:03.893555 3334 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet a95a3f25901e468ab6e6cd02c2cd035f (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]) successfully deleted
I20261118 02:55:03.894984 4341 ts_tablet_manager.cc:1480] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:03.885650 4317 tablet_replica.cc:333] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:03.894260 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "72f427c72a554ac1a0a2f2c3a78865da" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:03.896133 3616 raft_consensus.cc:2490] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:03.897914 4317 raft_consensus.cc:2261] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:03.898124 4317 raft_consensus.cc:2290] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:03.898994 4317 ts_tablet_manager.cc:1962] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:03.899885 4341 tablet_bootstrap.cc:492] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:03.901618 4341 tablet_bootstrap.cc:654] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:03.902745 3838 leader_election.cc:304] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters: 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:03.903264 4325 raft_consensus.cc:2826] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:03.903398 4325 raft_consensus.cc:499] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:03.903484 4325 raft_consensus.cc:3096] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.907677 4338 tablet_replica.cc:333] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:03.907611 4325 raft_consensus.cc:521] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:03.907984 4338 raft_consensus.cc:2261] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:03.908247 4338 raft_consensus.cc:2290] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:03.909160 3753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "72f427c72a554ac1a0a2f2c3a78865da" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:03.909413 3753 raft_consensus.cc:3096] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.910567 4338 ts_tablet_manager.cc:1962] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:03.912290 4325 leader_election.cc:290] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:03.912967 3753 raft_consensus.cc:2490] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:03.913532 3836 leader_election.cc:304] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:03.917097 4325 raft_consensus.cc:2826] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:03.917665 4325 raft_consensus.cc:705] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:03.918107 4325 consensus_queue.cc:238] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:03.918912 4341 tablet_bootstrap.cc:492] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:03.919047 4341 ts_tablet_manager.cc:1449] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20261118 02:55:03.919775 4341 raft_consensus.cc:360] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.919951 4341 raft_consensus.cc:386] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:03.920032 4341 raft_consensus.cc:748] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:03.920293 4341 consensus_queue.cc:261] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.920884 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "72f427c72a554ac1a0a2f2c3a78865da" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:03.921077 3616 raft_consensus.cc:3096] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:03.924714 3616 raft_consensus.cc:2490] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:03.925630 4317 ts_tablet_manager.cc:1975] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:03.925784 4317 log.cc:1199] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/2de409a48f5b488484d74008f186d166
I20261118 02:55:03.926209 4317 ts_tablet_manager.cc:1996] T 2de409a48f5b488484d74008f186d166 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:03.927208 4317 tablet_replica.cc:333] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:03.927430 4317 raft_consensus.cc:2261] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:03.927634 4317 raft_consensus.cc:2290] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:03.927621 3334 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet 2de409a48f5b488484d74008f186d166 (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]) successfully deleted
I20261118 02:55:03.927918 4341 ts_tablet_manager.cc:1480] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20261118 02:55:03.934151 4338 ts_tablet_manager.cc:1975] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:03.934402 4338 log.cc:1199] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/e9323addc63245bfacb3854782425347
I20261118 02:55:03.934832 4338 ts_tablet_manager.cc:1996] T e9323addc63245bfacb3854782425347 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:03.936023 3332 catalog_manager.cc:5065] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): tablet e9323addc63245bfacb3854782425347 (table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]) successfully deleted
I20261118 02:55:03.937003 4323 tablet_bootstrap.cc:492] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:03.937216 4323 ts_tablet_manager.cc:1449] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.003s
I20261118 02:55:03.938122 4323 raft_consensus.cc:360] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.938447 4323 raft_consensus.cc:386] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:03.938594 4323 raft_consensus.cc:748] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:03.938946 4323 consensus_queue.cc:261] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.939652 4215 consensus_queue.cc:1049] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.941550 4323 ts_tablet_manager.cc:1480] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20261118 02:55:03.944454 4317 ts_tablet_manager.cc:1962] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:03.955156 4317 ts_tablet_manager.cc:1975] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:03.955300 4317 log.cc:1199] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/55f81e7b62e54ec18c7c13a200edb68f
I20261118 02:55:03.955766 4317 ts_tablet_manager.cc:1996] T 55f81e7b62e54ec18c7c13a200edb68f P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:03.956708 4317 tablet_replica.cc:333] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:03.956945 4317 raft_consensus.cc:2261] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:03.957273 4317 raft_consensus.cc:2290] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:03.957788 3334 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet 55f81e7b62e54ec18c7c13a200edb68f (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]) successfully deleted
I20261118 02:55:03.958659 4325 consensus_queue.cc:1049] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:03.966035 4328 tablet_bootstrap.cc:492] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:03.966246 4328 ts_tablet_manager.cc:1449] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.083s user 0.018s sys 0.014s
I20261118 02:55:03.969515 4328 raft_consensus.cc:360] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.969734 4328 raft_consensus.cc:386] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:03.969825 4328 raft_consensus.cc:748] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:03.970053 4328 consensus_queue.cc:261] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:03.970542 4328 ts_tablet_manager.cc:1480] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20261118 02:55:03.971829 4317 ts_tablet_manager.cc:1962] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:03.974143 4328 tablet_bootstrap.cc:492] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:03.989940 4316 ts_tablet_manager.cc:1975] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:03.990080 4316 log.cc:1199] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/a95a3f25901e468ab6e6cd02c2cd035f
I20261118 02:55:03.991266 4328 tablet_bootstrap.cc:654] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:03.991525 4316 ts_tablet_manager.cc:1996] T a95a3f25901e468ab6e6cd02c2cd035f P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:03.992662 3333 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet a95a3f25901e468ab6e6cd02c2cd035f (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]) successfully deleted
I20261118 02:55:03.999418 4316 tablet_replica.cc:333] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:03.999660 4316 raft_consensus.cc:2261] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:04.000298 4316 raft_consensus.cc:2290] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:04.001791 4317 ts_tablet_manager.cc:1975] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:04.002020 4317 log.cc:1199] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/e9323addc63245bfacb3854782425347
I20261118 02:55:04.002501 4317 ts_tablet_manager.cc:1996] T e9323addc63245bfacb3854782425347 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:04.003826 3334 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet e9323addc63245bfacb3854782425347 (table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]) successfully deleted
I20261118 02:55:04.004763 4316 ts_tablet_manager.cc:1962] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:04.011252 4328 tablet_bootstrap.cc:492] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:04.011423 4328 ts_tablet_manager.cc:1449] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20261118 02:55:04.012310 4328 raft_consensus.cc:360] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.012499 4328 raft_consensus.cc:386] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.012580 4328 raft_consensus.cc:748] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.012864 4328 consensus_queue.cc:261] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.013844 4328 ts_tablet_manager.cc:1480] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:04.021203 3268 catalog_manager.cc:2772] deleted HMS catalog entry for table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]
I20261118 02:55:04.021359 4316 ts_tablet_manager.cc:1975] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:04.021528 4316 log.cc:1199] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/2de409a48f5b488484d74008f186d166
I20261118 02:55:04.021843 3268 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:52040 (request call id 33) took 3151 ms. Trace:
I20261118 02:55:04.021916 4316 ts_tablet_manager.cc:1996] T 2de409a48f5b488484d74008f186d166 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:04.021958 3268 rpcz_store.cc:276] 1118 02:55:00.869974 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:02.404597 (+1534623us) service_pool.cc:224] Handling call
1118 02:55:03.446961 (+1042364us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:04.021807 (+574846us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1569,"Hive Metastore.run_cpu_time_us":205,"Hive Metastore.run_wall_time_us":572184}
I20261118 02:55:04.022964 4316 tablet_replica.cc:333] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:04.023159 4316 raft_consensus.cc:2261] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:04.023161 3333 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet 2de409a48f5b488484d74008f186d166 (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]) successfully deleted
I20261118 02:55:04.023511 4316 raft_consensus.cc:2290] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:04.024121 4316 ts_tablet_manager.cc:1962] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:04.030874 4316 ts_tablet_manager.cc:1975] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:04.031024 4316 log.cc:1199] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/55f81e7b62e54ec18c7c13a200edb68f
I20261118 02:55:04.031443 4316 ts_tablet_manager.cc:1996] T 55f81e7b62e54ec18c7c13a200edb68f P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:04.032509 4316 tablet_replica.cc:333] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:04.033041 3333 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet 55f81e7b62e54ec18c7c13a200edb68f (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]) successfully deleted
I20261118 02:55:04.033195 4316 raft_consensus.cc:2261] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:04.033378 4316 raft_consensus.cc:2290] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:04.033366 3344 catalog_manager.cc:2215] created HMS catalog entry for table [id=a59b9425c01049269883cf7767936c67]
I20261118 02:55:04.036355 4316 ts_tablet_manager.cc:1962] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:04.057052 4316 ts_tablet_manager.cc:1975] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:04.057224 4316 log.cc:1199] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/e9323addc63245bfacb3854782425347
I20261118 02:55:04.057595 4316 ts_tablet_manager.cc:1996] T e9323addc63245bfacb3854782425347 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:04.067538 3597 tablet_service.cc:1500] Processing CreateTablet for tablet f094af4ac3604326a539461d06b01868 (DEFAULT_TABLE table=default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.068518 3734 tablet_service.cc:1500] Processing CreateTablet for tablet f094af4ac3604326a539461d06b01868 (DEFAULT_TABLE table=default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.069027 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f094af4ac3604326a539461d06b01868. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.072523 3880 tablet_service.cc:1500] Processing CreateTablet for tablet f094af4ac3604326a539461d06b01868 (DEFAULT_TABLE table=default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.073032 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f094af4ac3604326a539461d06b01868. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.076151 4265 raft_consensus.cc:499] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:04.076321 4265 raft_consensus.cc:521] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.068042 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f094af4ac3604326a539461d06b01868. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.077124 4265 leader_election.cc:290] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.078401 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "01778879aaa44ea9855fae67cfb3952d" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:04.078657 3752 raft_consensus.cc:2490] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:04.079171 3552 leader_election.cc:304] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:04.079885 4265 raft_consensus.cc:2826] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:04.079988 4265 raft_consensus.cc:499] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:04.080070 4265 raft_consensus.cc:3096] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.079994 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "01778879aaa44ea9855fae67cfb3952d" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:04.080575 3902 raft_consensus.cc:2490] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:04.081190 4325 raft_consensus.cc:499] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:04.081338 4325 raft_consensus.cc:521] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.081983 4325 leader_election.cc:290] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:04.082883 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2b01d7e596bd4790b8ba5dc88d8e12cc" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:04.082916 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2b01d7e596bd4790b8ba5dc88d8e12cc" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:04.083098 3616 raft_consensus.cc:2490] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:04.083117 3752 raft_consensus.cc:2490] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:04.083560 3836 leader_election.cc:304] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:04.083521 4265 raft_consensus.cc:521] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.084262 4325 raft_consensus.cc:2826] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:04.084379 4325 raft_consensus.cc:499] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:04.084477 4325 raft_consensus.cc:3096] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.084460 3752 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "01778879aaa44ea9855fae67cfb3952d" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:04.084643 3752 raft_consensus.cc:3096] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.087608 3752 raft_consensus.cc:2490] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:04.087842 4325 raft_consensus.cc:521] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.088083 3552 leader_election.cc:304] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:04.088405 4270 raft_consensus.cc:2826] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:04.088578 4270 raft_consensus.cc:705] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:04.088869 4270 consensus_queue.cc:238] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.090488 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2b01d7e596bd4790b8ba5dc88d8e12cc" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:04.090677 3616 raft_consensus.cc:3096] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.093236 3616 raft_consensus.cc:2490] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:04.093619 4341 tablet_bootstrap.cc:492] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:04.094107 3838 leader_election.cc:304] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:04.094774 4215 raft_consensus.cc:2826] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:04.094908 4215 raft_consensus.cc:705] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:04.095204 4215 consensus_queue.cc:238] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.095428 4341 tablet_bootstrap.cc:654] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.096323 4328 tablet_bootstrap.cc:492] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:04.096511 4323 tablet_bootstrap.cc:492] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:04.098119 4328 tablet_bootstrap.cc:654] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.098690 4323 tablet_bootstrap.cc:654] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.108286 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "01778879aaa44ea9855fae67cfb3952d" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:04.108562 3902 raft_consensus.cc:3096] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.109030 3753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2b01d7e596bd4790b8ba5dc88d8e12cc" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:04.109234 3753 raft_consensus.cc:3096] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.111523 3902 raft_consensus.cc:2490] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:04.112910 3753 raft_consensus.cc:2490] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:04.113003 4265 leader_election.cc:290] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.114075 4325 leader_election.cc:290] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:04.118269 3344 catalog_manager.cc:6393] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da] alter complete (version 1)
I20261118 02:55:04.120646 3344 catalog_manager.cc:5940] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705) does not have the latest schema for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]). Expected version 1 got 0
I20261118 02:55:04.121176 3344 catalog_manager.cc:5840] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } }
I20261118 02:55:04.122175 3344 catalog_manager.cc:5940] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705) does not have the latest schema for tablet 72f427c72a554ac1a0a2f2c3a78865da (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]). Expected version 1 got 0
I20261118 02:55:04.128289 3333 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet e9323addc63245bfacb3854782425347 (table default.table_bca0059ea2004174b89866f3e8cc68fb [id=469b750094124f159d835e6fb5a6486e]) successfully deleted
I20261118 02:55:04.132853 4328 tablet_bootstrap.cc:492] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:04.133052 4328 ts_tablet_manager.cc:1449] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.007s
I20261118 02:55:04.134003 4328 raft_consensus.cc:360] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.134217 4328 raft_consensus.cc:386] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.134294 4328 raft_consensus.cc:748] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.142890 4210 consensus_queue.cc:1049] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.151130 4215 consensus_queue.cc:1049] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.170456 4341 tablet_bootstrap.cc:492] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:04.170768 4341 ts_tablet_manager.cc:1449] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.003s
I20261118 02:55:04.173640 4341 raft_consensus.cc:360] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.173959 4341 raft_consensus.cc:386] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.175185 4323 tablet_bootstrap.cc:492] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:04.175429 4323 ts_tablet_manager.cc:1449] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.006s
I20261118 02:55:04.176498 4323 raft_consensus.cc:360] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.176769 4323 raft_consensus.cc:386] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.176898 4323 raft_consensus.cc:748] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.177419 4323 consensus_queue.cc:261] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.183075 4270 consensus_queue.cc:1049] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20261118 02:55:04.183395 4265 consensus_queue.cc:1049] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.174098 4341 raft_consensus.cc:748] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.196213 4323 ts_tablet_manager.cc:1480] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20261118 02:55:04.195526 4341 consensus_queue.cc:261] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.210675 4341 ts_tablet_manager.cc:1480] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20261118 02:55:04.221591 3344 catalog_manager.cc:6393] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b] alter complete (version 1)
W20261118 02:55:04.226326 3344 catalog_manager.cc:6040] Failed to send AlterTable RPC for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 01778879aaa44ea9855fae67cfb3952d consensus state has no leader
I20261118 02:55:04.232226 4328 consensus_queue.cc:261] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.232650 4333 tablet.cc:1759] T 72f427c72a554ac1a0a2f2c3a78865da P 3f1973986ec1466eab5bec01dd81654f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:04.232920 4328 ts_tablet_manager.cc:1480] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.100s user 0.002s sys 0.000s
I20261118 02:55:04.242697 4330 tablet.cc:1759] T 72f427c72a554ac1a0a2f2c3a78865da P 1fdf7d7ece324fefb3a0c376add5b7e8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:04.250113 4332 tablet.cc:1759] T 72f427c72a554ac1a0a2f2c3a78865da P 99a39b1d9d814deeb3a8a644e9b392ef: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:04.250375 4325 raft_consensus.cc:499] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:04.253129 4325 raft_consensus.cc:521] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.254190 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "f094af4ac3604326a539461d06b01868" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:04.254416 3616 raft_consensus.cc:2490] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:04.255401 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "f094af4ac3604326a539461d06b01868" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:04.255657 3751 raft_consensus.cc:2490] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:04.256052 3838 leader_election.cc:304] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:04.256448 4210 raft_consensus.cc:2826] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:04.256536 4325 leader_election.cc:290] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:04.256567 4210 raft_consensus.cc:499] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:04.256664 4210 raft_consensus.cc:3096] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.259512 4210 raft_consensus.cc:521] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.260694 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "f094af4ac3604326a539461d06b01868" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:04.260905 3616 raft_consensus.cc:3096] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.253939 3344 catalog_manager.cc:5940] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377) does not have the latest schema for tablet 72f427c72a554ac1a0a2f2c3a78865da (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]). Expected version 1 got 0
I20261118 02:55:04.261639 4210 leader_election.cc:290] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:04.262578 3344 catalog_manager.cc:5940] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377) does not have the latest schema for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]). Expected version 1 got 0
I20261118 02:55:04.263835 3616 raft_consensus.cc:2490] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:04.264951 3838 leader_election.cc:304] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:04.265004 3347 catalog_manager.cc:2215] created HMS catalog entry for table [id=e797fad7075b40d9bdd674a0e0023139]
I20261118 02:55:04.265265 4210 raft_consensus.cc:2826] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:04.265422 4210 raft_consensus.cc:705] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:04.265718 4210 consensus_queue.cc:238] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.267365 3270 catalog_manager.cc:2215] created HMS catalog entry for table [id=baa54e879f6246509f1c5ca1876f0b25]
W20261118 02:55:04.268260 3270 catalog_manager.cc:2243] Illegal state: an error occurred while writing to the sys-catalog: replica 26c6b20fa203404a88463ea27bdd3c7d is not leader of this config: current role FOLLOWER
I20261118 02:55:04.269680 3344 catalog_manager.cc:5940] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237) does not have the latest schema for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]). Expected version 1 got 0
I20261118 02:55:04.270011 3344 catalog_manager.cc:5940] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237) does not have the latest schema for tablet 72f427c72a554ac1a0a2f2c3a78865da (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]). Expected version 1 got 0
I20261118 02:55:04.270249 3344 catalog_manager.cc:5840] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: HEALTHY } } }
W20261118 02:55:04.275611 3341 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:04.275942 3343 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_30b09a2798a54b01be7618b8a28eafaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:04.276232 3345 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_38304cbb980440c2a8b8a6c13f49949c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:04.277308 3340 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6e09d0e89581458183ed524cc1404849 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:04.277547 3338 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d4fec0aaed15433ea373740872db53d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 02:55:04.281633 3334 catalog_manager.cc:4847] Async tablet task AlterTable RPC for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 01778879aaa44ea9855fae67cfb3952d consensus state has no leader
I20261118 02:55:04.297396 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "f094af4ac3604326a539461d06b01868" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:04.298947 3751 raft_consensus.cc:3096] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.302254 3751 raft_consensus.cc:2490] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:04.312664 3880 tablet_service.cc:1500] Processing CreateTablet for tablet b610080fc91745c7be260699946d5681 (DEFAULT_TABLE table=default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.313160 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b610080fc91745c7be260699946d5681. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.318404 3347 catalog_manager.cc:5840] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: UNKNOWN } } }
I20261118 02:55:04.318907 3347 catalog_manager.cc:5940] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705) does not have the latest schema for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]). Expected version 1 got 0
I20261118 02:55:04.320631 3734 tablet_service.cc:1500] Processing CreateTablet for tablet b610080fc91745c7be260699946d5681 (DEFAULT_TABLE table=default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.321077 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b610080fc91745c7be260699946d5681. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.325342 3597 tablet_service.cc:1500] Processing CreateTablet for tablet b610080fc91745c7be260699946d5681 (DEFAULT_TABLE table=default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.325804 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b610080fc91745c7be260699946d5681. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.327931 4210 consensus_queue.cc:1049] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.328421 4210 consensus_queue.cc:1049] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.331703 3347 catalog_manager.cc:5940] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237) does not have the latest schema for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]). Expected version 1 got 0
I20261118 02:55:04.332062 3347 catalog_manager.cc:5840] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f1973986ec1466eab5bec01dd81654f (127.2.229.67). New cstate: current_term: 1 leader_uuid: "3f1973986ec1466eab5bec01dd81654f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: HEALTHY } } }
I20261118 02:55:04.334553 4323 tablet_bootstrap.cc:492] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:04.336999 4323 tablet_bootstrap.cc:654] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.338425 4328 tablet_bootstrap.cc:492] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:04.338806 4341 tablet_bootstrap.cc:492] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:04.340728 4341 tablet_bootstrap.cc:654] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.344504 4328 tablet_bootstrap.cc:654] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.344513 3344 catalog_manager.cc:5940] TS 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377) does not have the latest schema for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]). Expected version 1 got 0
I20261118 02:55:04.349254 4323 tablet_bootstrap.cc:492] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:04.349424 4323 ts_tablet_manager.cc:1449] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20261118 02:55:04.350948 3347 catalog_manager.cc:6393] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b] alter complete (version 1)
I20261118 02:55:04.350237 4323 raft_consensus.cc:360] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.351199 4323 raft_consensus.cc:386] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.351294 4323 raft_consensus.cc:748] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.351647 3347 master_service.cc:965] ReplaceTablet: received request to replace tablet e9323addc63245bfacb3854782425347 from {username='slave'} at 127.0.0.1:48092
I20261118 02:55:04.351619 4323 consensus_queue.cc:261] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.352317 4323 ts_tablet_manager.cc:1480] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20261118 02:55:04.352272 3347 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48092:
name: "default.Table_e39f59e2fdbb47fda4c10beba29ad8f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:04.358325 3344 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48092:
name: "default.Table_fd4fcac1f1d946fa898176f1d5e87e9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:04.377179 4341 tablet_bootstrap.cc:492] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:04.377388 4341 ts_tablet_manager.cc:1449] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.000s
I20261118 02:55:04.378207 4341 raft_consensus.cc:360] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.378402 4341 raft_consensus.cc:386] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.378512 4341 raft_consensus.cc:748] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.379442 4328 tablet_bootstrap.cc:492] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:04.379642 4328 ts_tablet_manager.cc:1449] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.003s
I20261118 02:55:04.380507 4328 raft_consensus.cc:360] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.380713 4328 raft_consensus.cc:386] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.380808 4328 raft_consensus.cc:748] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.381094 4328 consensus_queue.cc:261] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.385006 4328 ts_tablet_manager.cc:1480] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 02:55:04.406440 4341 consensus_queue.cc:261] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.413273 4341 ts_tablet_manager.cc:1480] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.036s user 0.002s sys 0.001s
I20261118 02:55:04.417950 3341 catalog_manager.cc:2215] created HMS catalog entry for table [id=e0ed3867fcbb42a4b0133f38e0372a6f]
I20261118 02:55:04.425582 4330 tablet.cc:1759] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:04.430912 3341 catalog_manager.cc:5940] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705) does not have the latest schema for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]). Expected version 1 got 0
I20261118 02:55:04.431914 4332 tablet.cc:1759] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:04.438508 4333 tablet.cc:1759] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20261118 02:55:04.444600 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 1904140da78b4567a9decb250c6f74a3 (DEFAULT_TABLE table=default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.444877 3597 tablet_service.cc:1500] Processing CreateTablet for tablet 1904140da78b4567a9decb250c6f74a3 (DEFAULT_TABLE table=default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.445073 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1904140da78b4567a9decb250c6f74a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.445312 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1904140da78b4567a9decb250c6f74a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.448736 3880 tablet_service.cc:1500] Processing CreateTablet for tablet 1904140da78b4567a9decb250c6f74a3 (DEFAULT_TABLE table=default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.449203 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1904140da78b4567a9decb250c6f74a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.450577 3341 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48092:
table { table_name: "default.Table_4f7796d5d5d54def843b971e66d15ea6" } new_table_name: "default.Table_79eb11797e3f4bc2bffe161096300f7a" modify_external_catalogs: true
I20261118 02:55:04.453258 4323 tablet_bootstrap.cc:492] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:04.455107 4323 tablet_bootstrap.cc:654] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.457552 3271 catalog_manager.cc:2772] deleted HMS catalog entry for table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]
I20261118 02:55:04.458042 3271 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:52040 (request call id 27) took 3723 ms. Trace:
I20261118 02:55:04.458174 3271 rpcz_store.cc:276] 1118 02:55:00.734475 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:02.105423 (+1370948us) service_pool.cc:224] Handling call
1118 02:55:03.452696 (+1347273us) catalog_manager.cc:2434] Looking up, locking, and authorizing table
1118 02:55:04.458000 (+1005304us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":814321,"Hive Metastore.run_cpu_time_us":198,"Hive Metastore.run_wall_time_us":189927}
I20261118 02:55:04.469877 4341 tablet_bootstrap.cc:492] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:04.471861 4341 tablet_bootstrap.cc:654] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.481155 4328 tablet_bootstrap.cc:492] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:04.483292 4328 tablet_bootstrap.cc:654] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.491559 4328 tablet_bootstrap.cc:492] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:04.491778 4328 ts_tablet_manager.cc:1449] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20261118 02:55:04.492671 4328 raft_consensus.cc:360] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.492899 4328 raft_consensus.cc:386] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.492985 4328 raft_consensus.cc:748] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.494174 4341 tablet_bootstrap.cc:492] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:04.494375 4341 ts_tablet_manager.cc:1449] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.004s
I20261118 02:55:04.494832 3343 catalog_manager.cc:2215] created HMS catalog entry for table [id=dd0f01caa36a4d3e9795daaf698b1ce1]
I20261118 02:55:04.495195 4328 consensus_queue.cc:261] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.496940 4328 ts_tablet_manager.cc:1480] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20261118 02:55:04.497500 4341 raft_consensus.cc:360] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.497733 4341 raft_consensus.cc:386] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.503237 4341 raft_consensus.cc:748] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.503867 4341 consensus_queue.cc:261] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.508061 4341 ts_tablet_manager.cc:1480] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20261118 02:55:04.511054 4323 tablet_bootstrap.cc:492] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:04.511263 4323 ts_tablet_manager.cc:1449] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.000s
I20261118 02:55:04.512176 4323 raft_consensus.cc:360] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.512821 4323 raft_consensus.cc:386] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.512908 4323 raft_consensus.cc:748] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.513221 4323 consensus_queue.cc:261] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.516052 4323 ts_tablet_manager.cc:1480] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20261118 02:55:04.517192 3343 catalog_manager.cc:6393] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7] alter complete (version 1)
I20261118 02:55:04.524865 3734 tablet_service.cc:1500] Processing CreateTablet for tablet b129518ce23c45758ebcaa55410757d8 (DEFAULT_TABLE table=default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.525425 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b129518ce23c45758ebcaa55410757d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.525584 3597 tablet_service.cc:1500] Processing CreateTablet for tablet b129518ce23c45758ebcaa55410757d8 (DEFAULT_TABLE table=default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.526010 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b129518ce23c45758ebcaa55410757d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.528980 3880 tablet_service.cc:1500] Processing CreateTablet for tablet b129518ce23c45758ebcaa55410757d8 (DEFAULT_TABLE table=default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.529498 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b129518ce23c45758ebcaa55410757d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.535542 4323 tablet_bootstrap.cc:492] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:04.537458 4323 tablet_bootstrap.cc:654] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.542994 3343 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48092:
table { table_name: "default.Table_0385cef6593c419eb426657c8cbef2bf" } new_table_name: "default.Table_496441d99faa4e2b8c7cd98e4a5ce745" modify_external_catalogs: true
I20261118 02:55:04.548003 4341 tablet_bootstrap.cc:492] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:04.549827 4341 tablet_bootstrap.cc:654] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.551963 2965 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task8fD3U4/build/asan/bin/kudu pid 3246
I20261118 02:55:04.556473 3307 generic_service.cc:196] Checking for leaks (request via RPC)
I20261118 02:55:04.560854 4323 tablet_bootstrap.cc:492] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:04.561024 4323 ts_tablet_manager.cc:1449] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20261118 02:55:04.561877 4323 raft_consensus.cc:360] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.562078 4323 raft_consensus.cc:386] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.562173 4323 raft_consensus.cc:748] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.562422 4323 consensus_queue.cc:261] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.564137 4323 ts_tablet_manager.cc:1480] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20261118 02:55:04.578416 4328 tablet_bootstrap.cc:492] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:04.580322 3345 catalog_manager.cc:2215] created HMS catalog entry for table [id=7140b942170b4483b0e01e32d5ba5e6d]
I20261118 02:55:04.584175 4328 tablet_bootstrap.cc:654] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.588632 3345 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48092 (request call id 15) took 1009 ms. Trace:
I20261118 02:55:04.588814 3345 rpcz_store.cc:276] 1118 02:55:03.579044 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.579563 (+ 519us) service_pool.cc:224] Handling call
1118 02:55:04.276321 (+696758us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:04.276726 (+ 405us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:04.580256 (+303530us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:04.588342 (+ 8086us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:04.588462 (+ 120us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:04.588586 (+ 124us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:04.581268 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:04.581427 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:04.581434 (+ 7us) write_op.cc:435] Acquired schema lock
1118 02:55:04.581437 (+ 3us) tablet.cc:670] Decoding operations
1118 02:55:04.582158 (+ 721us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:04.582175 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:04.582181 (+ 6us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:04.582277 (+ 96us) tablet.cc:709] Row locks acquired
1118 02:55:04.582280 (+ 3us) write_op.cc:260] PREPARE: finished
1118 02:55:04.582325 (+ 45us) write_op.cc:270] Start()
1118 02:55:04.582364 (+ 39us) write_op.cc:276] Timestamp: P: 1794970504582211 usec, L: 0
1118 02:55:04.582368 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:04.582610 (+ 242us) log.cc:844] Serialized 432 byte log entry
1118 02:55:04.586943 (+ 4333us) op_driver.cc:466] REPLICATION: finished
1118 02:55:04.587060 (+ 117us) write_op.cc:301] APPLY: starting
1118 02:55:04.587079 (+ 19us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:04.587126 (+ 47us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:04.587163 (+ 37us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:04.587299 (+ 136us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:04.587333 (+ 34us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:04.587341 (+ 8us) write_op.cc:312] APPLY: finished
1118 02:55:04.587424 (+ 83us) log.cc:844] Serialized 28 byte log entry
1118 02:55:04.587491 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:04.587515 (+ 24us) write_op.cc:454] Released schema lock
1118 02:55:04.587522 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:04.587594 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:04.587599 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":217932,"Hive Metastore.run_cpu_time_us":329,"Hive Metastore.run_wall_time_us":83894,"child_traces":[["op",{"apply.queue_time_us":65,"apply.run_cpu_time_us":525,"apply.run_wall_time_us":524,"num_ops":2,"prepare.queue_time_us":162,"prepare.run_cpu_time_us":936,"prepare.run_wall_time_us":1496,"raft.queue_time_us":515,"raft.run_cpu_time_us":686,"raft.run_wall_time_us":684,"replication_time_us":4540}]]}
I20261118 02:55:04.605414 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 2c92955aa65142f1bd20f4b8f3321500 (DEFAULT_TABLE table=default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.605541 3597 tablet_service.cc:1500] Processing CreateTablet for tablet 2c92955aa65142f1bd20f4b8f3321500 (DEFAULT_TABLE table=default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.605924 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c92955aa65142f1bd20f4b8f3321500. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.605921 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c92955aa65142f1bd20f4b8f3321500. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.609285 3880 tablet_service.cc:1500] Processing CreateTablet for tablet 2c92955aa65142f1bd20f4b8f3321500 (DEFAULT_TABLE table=default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:04.609778 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c92955aa65142f1bd20f4b8f3321500. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:04.612345 4323 tablet_bootstrap.cc:492] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:04.614121 4323 tablet_bootstrap.cc:654] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.618650 3345 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48092:
table { table_name: "default.Table_a7f3710d39fb447da94e16d4ab3933fd" } modify_external_catalogs: true
I20261118 02:55:04.618996 3345 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48092:
table { table_name: "default.Table_a7f3710d39fb447da94e16d4ab3933fd" } modify_external_catalogs: true
I20261118 02:55:04.627017 4323 tablet_bootstrap.cc:492] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:04.627230 4323 ts_tablet_manager.cc:1449] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20261118 02:55:04.627993 4323 raft_consensus.cc:360] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.628306 4323 raft_consensus.cc:386] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.628427 4323 raft_consensus.cc:748] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.628737 4323 consensus_queue.cc:261] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.631083 4328 tablet_bootstrap.cc:492] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:04.632825 4323 ts_tablet_manager.cc:1480] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 02:55:04.633181 4328 ts_tablet_manager.cc:1449] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.003s
I20261118 02:55:04.634070 4328 raft_consensus.cc:360] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.634274 4328 raft_consensus.cc:386] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.634378 4328 raft_consensus.cc:748] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.637547 4341 tablet_bootstrap.cc:492] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:04.637727 4341 ts_tablet_manager.cc:1449] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.006s
I20261118 02:55:04.638218 4328 consensus_queue.cc:261] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.638607 4341 raft_consensus.cc:360] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.638782 4341 raft_consensus.cc:386] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.638867 4341 raft_consensus.cc:748] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.639106 4341 consensus_queue.cc:261] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.646626 4328 ts_tablet_manager.cc:1480] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20261118 02:55:04.650271 4341 ts_tablet_manager.cc:1480] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20261118 02:55:04.650406 4328 tablet_bootstrap.cc:492] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:04.650604 4341 tablet_bootstrap.cc:492] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:04.652341 4341 tablet_bootstrap.cc:654] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.653347 4360 raft_consensus.cc:499] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:04.653501 4360 raft_consensus.cc:521] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.654232 4360 leader_election.cc:290] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.655382 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b610080fc91745c7be260699946d5681" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:04.655633 3616 raft_consensus.cc:2490] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:04.656010 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b610080fc91745c7be260699946d5681" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:04.656296 3902 raft_consensus.cc:2490] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:04.657063 3690 leader_election.cc:304] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:04.657759 4360 raft_consensus.cc:2826] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:04.657883 4360 raft_consensus.cc:499] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:04.657971 4360 raft_consensus.cc:3096] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.661325 4328 tablet_bootstrap.cc:654] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:04.662608 4360 raft_consensus.cc:521] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.663757 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b610080fc91745c7be260699946d5681" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:04.663996 3616 raft_consensus.cc:3096] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.667074 3616 raft_consensus.cc:2490] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:04.667618 3691 leader_election.cc:304] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:04.667917 4300 raft_consensus.cc:2826] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:04.671132 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b610080fc91745c7be260699946d5681" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:04.671389 3902 raft_consensus.cc:3096] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.674564 3902 raft_consensus.cc:2490] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:04.675848 4360 leader_election.cc:290] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.681222 4328 tablet_bootstrap.cc:492] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:04.681438 4328 ts_tablet_manager.cc:1449] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.004s
I20261118 02:55:04.682225 4328 raft_consensus.cc:360] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.682440 4328 raft_consensus.cc:386] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.682519 4328 raft_consensus.cc:748] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.682826 4328 consensus_queue.cc:261] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.683494 4328 ts_tablet_manager.cc:1480] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20261118 02:55:04.691319 4300 raft_consensus.cc:705] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:04.691655 4300 consensus_queue.cc:238] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:04.695037 4341 tablet_bootstrap.cc:492] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:04.703747 4360 consensus_queue.cc:1049] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.708168 4300 consensus_queue.cc:1049] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.724174 4263 raft_consensus.cc:499] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:04.724373 4263 raft_consensus.cc:521] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.724961 4263 leader_election.cc:290] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.725039 4341 ts_tablet_manager.cc:1449] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20261118 02:55:04.726269 3753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b129518ce23c45758ebcaa55410757d8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:04.726512 3753 raft_consensus.cc:2490] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:04.727325 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b129518ce23c45758ebcaa55410757d8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:04.727535 3902 raft_consensus.cc:2490] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:04.728065 3552 leader_election.cc:304] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:04.728480 4341 raft_consensus.cc:360] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.728674 4341 raft_consensus.cc:386] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:04.728690 4265 raft_consensus.cc:2826] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:04.728730 4341 raft_consensus.cc:748] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:04.728797 4265 raft_consensus.cc:499] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:04.728873 4265 raft_consensus.cc:3096] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.728973 4341 consensus_queue.cc:261] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.730585 4341 ts_tablet_manager.cc:1480] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20261118 02:55:04.731884 4265 raft_consensus.cc:521] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.735399 4265 leader_election.cc:290] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.736183 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b129518ce23c45758ebcaa55410757d8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:04.736174 3753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b129518ce23c45758ebcaa55410757d8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:04.736389 3753 raft_consensus.cc:3096] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.736858 3902 raft_consensus.cc:3096] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.739348 3753 raft_consensus.cc:2490] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:04.739645 3902 raft_consensus.cc:2490] T b129518ce23c45758ebcaa55410757d8 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:04.740029 3553 leader_election.cc:304] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:04.743608 4265 raft_consensus.cc:2826] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:04.744136 4265 raft_consensus.cc:705] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:04.744565 4265 consensus_queue.cc:238] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.768595 4263 consensus_queue.cc:1049] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.777444 4384 consensus_queue.cc:1049] T b129518ce23c45758ebcaa55410757d8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.793957 4384 raft_consensus.cc:499] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:04.794260 4384 raft_consensus.cc:521] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.795543 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1904140da78b4567a9decb250c6f74a3" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:04.795845 3751 raft_consensus.cc:2490] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:04.795965 4384 leader_election.cc:290] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.796357 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1904140da78b4567a9decb250c6f74a3" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:04.796380 3552 leader_election.cc:304] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:04.796587 3901 raft_consensus.cc:2490] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:04.796798 4265 raft_consensus.cc:2826] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:04.796900 4265 raft_consensus.cc:499] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:04.796975 4265 raft_consensus.cc:3096] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.799749 4265 raft_consensus.cc:521] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.800417 4265 leader_election.cc:290] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.801316 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1904140da78b4567a9decb250c6f74a3" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:04.801558 3751 raft_consensus.cc:3096] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.804633 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1904140da78b4567a9decb250c6f74a3" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:04.804874 3901 raft_consensus.cc:3096] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.809340 3751 raft_consensus.cc:2490] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:04.810060 3901 raft_consensus.cc:2490] T 1904140da78b4567a9decb250c6f74a3 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:04.810653 3553 leader_election.cc:304] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:04.810959 4387 raft_consensus.cc:2826] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:04.811206 4387 raft_consensus.cc:705] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:04.811496 4387 consensus_queue.cc:238] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.839188 4263 consensus_queue.cc:1049] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.840451 4265 consensus_queue.cc:1049] T 1904140da78b4567a9decb250c6f74a3 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.883309 3270 catalog_manager.cc:2225] deleted orphaned HMS catalog entry for table [id=baa54e879f6246509f1c5ca1876f0b25]
I20261118 02:55:04.883800 3270 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52040 (request call id 30) took 4046 ms. Trace:
I20261118 02:55:04.883934 3270 rpcz_store.cc:276] 1118 02:55:00.836975 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:02.367849 (+1530874us) service_pool.cc:224] Handling call
1118 02:55:03.447690 (+1079841us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:03.448571 (+ 881us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:04.267317 (+818746us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:04.268119 (+ 802us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:04.268123 (+ 4us) write_op.cc:454] Released schema lock
1118 02:55:04.268355 (+ 232us) catalog_manager.cc:2222] Rolling back HMS table creation
1118 02:55:04.883754 (+615399us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:04.268000 (+ 0us) op_driver.cc:391] HandleFailure(Illegal state: replica 26c6b20fa203404a88463ea27bdd3c7d is not leader of this config: current role FOLLOWER)
1118 02:55:04.268014 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:04.268022 (+ 8us) write_op.cc:454] Released schema lock
1118 02:55:04.268026 (+ 4us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":869586,"Hive Metastore.run_cpu_time_us":473,"Hive Metastore.run_wall_time_us":563445,"child_traces":[["op",{}]]}
I20261118 02:55:04.884406 3273 catalog_manager.cc:2772] deleted HMS catalog entry for table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]
W20261118 02:55:04.886116 3956 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.229.125:38339)
I20261118 02:55:04.889158 2965 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task8fD3U4/build/asan/bin/kudu with pid 3246
I20261118 02:55:04.893725 3342 catalog_manager.cc:3543] renamed table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6] in HMS: new name default.table_5091d82298ca44bea5335d4989eb02d8
W20261118 02:55:04.921594 3956 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.229.125:38339)
W20261118 02:55:04.923596 3333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.229.126:38025 (error 108) [suppressed 5 similar messages]
W20261118 02:55:04.934947 3333 consensus_peers.cc:597] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b -> Peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Couldn't send request to peer 26c6b20fa203404a88463ea27bdd3c7d. Status: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:04.936760 4384 raft_consensus.cc:499] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:04.936966 4384 raft_consensus.cc:521] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.937793 4384 leader_election.cc:290] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.938542 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2c92955aa65142f1bd20f4b8f3321500" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:04.938555 3753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2c92955aa65142f1bd20f4b8f3321500" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:04.938763 3902 raft_consensus.cc:2490] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:04.938771 3753 raft_consensus.cc:2490] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:04.939054 3339 catalog_manager.cc:2772] deleted HMS catalog entry for table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]
I20261118 02:55:04.939286 3552 leader_election.cc:304] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:04.939589 4384 raft_consensus.cc:2826] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:04.939703 4384 raft_consensus.cc:499] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:04.939790 4384 raft_consensus.cc:3096] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.942586 4384 raft_consensus.cc:521] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.943732 3753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2c92955aa65142f1bd20f4b8f3321500" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:04.943993 3753 raft_consensus.cc:3096] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.947000 3753 raft_consensus.cc:2490] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:04.947556 3552 leader_election.cc:304] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:04.947909 4263 raft_consensus.cc:2826] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:04.948065 4263 raft_consensus.cc:705] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:04.948397 4263 consensus_queue.cc:238] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:04.958894 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "2c92955aa65142f1bd20f4b8f3321500" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:04.959163 3902 raft_consensus.cc:3096] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:04.962152 3902 raft_consensus.cc:2490] T 2c92955aa65142f1bd20f4b8f3321500 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:04.963047 4384 leader_election.cc:290] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:04.978030 4387 consensus_queue.cc:1049] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:04.981541 4387 consensus_queue.cc:1049] T 2c92955aa65142f1bd20f4b8f3321500 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:05.040940 3346 catalog_manager.cc:3543] renamed table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73] in HMS: new name default.table_324a2600fef947748ef03e27f7cb7e09
I20261118 02:55:05.073117 3338 catalog_manager.cc:2215] created HMS catalog entry for table [id=00713c8ca8db4bf9a860d32614f99190]
I20261118 02:55:05.078691 3338 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48092 (request call id 23) took 1396 ms. Trace:
I20261118 02:55:05.078833 3338 rpcz_store.cc:276] 1118 02:55:03.682580 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.682977 (+ 397us) service_pool.cc:224] Handling call
1118 02:55:04.277647 (+594670us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:04.278070 (+ 423us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:05.073076 (+795006us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:05.078330 (+ 5254us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:05.078458 (+ 128us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:05.078602 (+ 144us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:05.073924 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:05.074068 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:05.074074 (+ 6us) write_op.cc:435] Acquired schema lock
1118 02:55:05.074075 (+ 1us) tablet.cc:670] Decoding operations
1118 02:55:05.074159 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:05.074171 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:05.074175 (+ 4us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:05.074242 (+ 67us) tablet.cc:709] Row locks acquired
1118 02:55:05.074244 (+ 2us) write_op.cc:260] PREPARE: finished
1118 02:55:05.074279 (+ 35us) write_op.cc:270] Start()
1118 02:55:05.074310 (+ 31us) write_op.cc:276] Timestamp: P: 1794970505074170 usec, L: 0
1118 02:55:05.074312 (+ 2us) op_driver.cc:350] REPLICATION: starting
1118 02:55:05.074493 (+ 181us) log.cc:844] Serialized 432 byte log entry
1118 02:55:05.077540 (+ 3047us) op_driver.cc:466] REPLICATION: finished
1118 02:55:05.077708 (+ 168us) write_op.cc:301] APPLY: starting
1118 02:55:05.077726 (+ 18us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:05.077778 (+ 52us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:05.077809 (+ 31us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:05.077946 (+ 137us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:05.077984 (+ 38us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:05.077992 (+ 8us) write_op.cc:312] APPLY: finished
1118 02:55:05.078117 (+ 125us) log.cc:844] Serialized 28 byte log entry
1118 02:55:05.078202 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:05.078226 (+ 24us) write_op.cc:454] Released schema lock
1118 02:55:05.078234 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:05.078316 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:05.078320 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":762713,"Hive Metastore.run_cpu_time_us":341,"Hive Metastore.run_wall_time_us":32084,"child_traces":[["op",{"apply.queue_time_us":114,"apply.run_cpu_time_us":599,"apply.run_wall_time_us":598,"num_ops":2,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":694,"prepare.run_wall_time_us":694,"raft.queue_time_us":93,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":332,"replication_time_us":3195}]]}
I20261118 02:55:05.103708 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 0e901a03b90e4c978eb98be87163cfe5 (DEFAULT_TABLE table=default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:05.103943 3880 tablet_service.cc:1500] Processing CreateTablet for tablet 0e901a03b90e4c978eb98be87163cfe5 (DEFAULT_TABLE table=default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:05.104224 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e901a03b90e4c978eb98be87163cfe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:05.104449 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e901a03b90e4c978eb98be87163cfe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:05.105386 3338 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48092:
table { table_name: "default.Table_2e3917ff98f2474984580e61caa9c17b" } new_table_name: "default.Table_3c225da1f7fa435083909e053ec9751a" modify_external_catalogs: true
I20261118 02:55:05.108930 3597 tablet_service.cc:1500] Processing CreateTablet for tablet 0e901a03b90e4c978eb98be87163cfe5 (DEFAULT_TABLE table=default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:05.109357 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e901a03b90e4c978eb98be87163cfe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:05.112860 4341 tablet_bootstrap.cc:492] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:05.114666 4341 tablet_bootstrap.cc:654] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:05.116966 4323 tablet_bootstrap.cc:492] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:05.118590 4323 tablet_bootstrap.cc:654] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:05.119942 4328 tablet_bootstrap.cc:492] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:05.121654 4328 tablet_bootstrap.cc:654] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:05.125878 4341 tablet_bootstrap.cc:492] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:05.126117 4341 ts_tablet_manager.cc:1449] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20261118 02:55:05.126958 4341 raft_consensus.cc:360] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.127720 4341 raft_consensus.cc:386] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:05.128005 4341 raft_consensus.cc:748] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:05.128612 4341 consensus_queue.cc:261] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.129287 3340 catalog_manager.cc:2215] created HMS catalog entry for table [id=01b05aad48954b8fb2aa6d7b31a6578c]
I20261118 02:55:05.131323 4328 tablet_bootstrap.cc:492] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:05.131340 4323 tablet_bootstrap.cc:492] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:05.131479 4323 ts_tablet_manager.cc:1449] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20261118 02:55:05.131623 4328 ts_tablet_manager.cc:1449] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20261118 02:55:05.133008 4328 raft_consensus.cc:360] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.133245 4328 raft_consensus.cc:386] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:05.134723 4323 raft_consensus.cc:360] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.134918 4323 raft_consensus.cc:386] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:05.135020 4323 raft_consensus.cc:748] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:05.135336 4323 consensus_queue.cc:261] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.136013 4323 ts_tablet_manager.cc:1480] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20261118 02:55:05.137785 3340 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48092 (request call id 11) took 1577 ms. Trace:
I20261118 02:55:05.137940 3340 rpcz_store.cc:276] 1118 02:55:03.560333 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:03.560379 (+ 46us) service_pool.cc:224] Handling call
1118 02:55:04.277424 (+717045us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 02:55:04.278168 (+ 744us) catalog_manager.cc:2188] Created new table and tablet info
1118 02:55:05.129224 (+851056us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 02:55:05.137471 (+ 8247us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 02:55:05.137615 (+ 144us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 02:55:05.137735 (+ 120us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:05.130158 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:05.130287 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:05.130294 (+ 7us) write_op.cc:435] Acquired schema lock
1118 02:55:05.130296 (+ 2us) tablet.cc:670] Decoding operations
1118 02:55:05.130394 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:05.130408 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:05.130412 (+ 4us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:05.130501 (+ 89us) tablet.cc:709] Row locks acquired
1118 02:55:05.130504 (+ 3us) write_op.cc:260] PREPARE: finished
1118 02:55:05.130542 (+ 38us) write_op.cc:270] Start()
1118 02:55:05.130577 (+ 35us) write_op.cc:276] Timestamp: P: 1794970505130524 usec, L: 0
1118 02:55:05.130581 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:05.130774 (+ 193us) log.cc:844] Serialized 432 byte log entry
1118 02:55:05.136311 (+ 5537us) op_driver.cc:466] REPLICATION: finished
1118 02:55:05.136956 (+ 645us) write_op.cc:301] APPLY: starting
1118 02:55:05.136974 (+ 18us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:05.137013 (+ 39us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:05.137046 (+ 33us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:05.137158 (+ 112us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:05.137183 (+ 25us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 02:55:05.137188 (+ 5us) write_op.cc:312] APPLY: finished
1118 02:55:05.137258 (+ 70us) log.cc:844] Serialized 28 byte log entry
1118 02:55:05.137319 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:05.137339 (+ 20us) write_op.cc:454] Released schema lock
1118 02:55:05.137346 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:05.137413 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:05.137417 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":794789,"Hive Metastore.run_cpu_time_us":277,"Hive Metastore.run_wall_time_us":56109,"child_traces":[["op",{"apply.queue_time_us":577,"apply.run_cpu_time_us":456,"apply.run_wall_time_us":457,"num_ops":2,"prepare.queue_time_us":202,"prepare.run_cpu_time_us":734,"prepare.run_wall_time_us":735,"raft.queue_time_us":1793,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":346,"replication_time_us":5697}]]}
I20261118 02:55:05.133363 4328 raft_consensus.cc:748] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:05.139240 4328 consensus_queue.cc:261] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
F20261118 02:55:05.142462 3527 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 21, previous 21; dumping first 2 out of 15 received events: NotificationEvent(eventId=21, eventTime=1794970503, eventType=CREATE_TABLE, dbName=default, tableName=table_f0ae490a0df64e1097e144c7a370df61, message={"server":"","servicePrincipal":"","db":"default","table":"table_f0ae490a0df64e1097e144c7a370df61","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f0ae490a0df64e1097e144c7a370df61\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970503},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_f0ae490a0df64e1097e144c7a370df61\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f0ae490a0df64e1097e144c7a370df61\",\"kudu.table_id\":\"a59b9425c01049269883cf7767936c67\",\"transient_lastDdlTime\":\"1794970503\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1794970503, eventType=DROP_TABLE, dbName=default, tableName=table_c113c2b57e17479e92e241234a241af7, message={"server":"","servicePrincipal":"","db":"default","table":"table_c113c2b57e17479e92e241234a241af7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c113c2b57e17479e92e241234a241af7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970500},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_c113c2b57e17479e92e241234a241af7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c113c2b57e17479e92e241234a241af7\",\"kudu.table_id\":\"da0cb25ccfed426ca61f0dd9f93d30ec\",\"transient_lastDdlTime\":\"1794970500\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20261118 02:55:05.144402 4328 ts_tablet_manager.cc:1480] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
@ 0x7f72ad19bd5d google::LogMessage::Fail() at ??:0
I20261118 02:55:05.145990 4341 ts_tablet_manager.cc:1480] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20261118 02:55:05.146613 3340 catalog_manager.cc:5840] T b129518ce23c45758ebcaa55410757d8 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } }
I20261118 02:55:05.147181 3340 catalog_manager.cc:5840] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66). New cstate: current_term: 1 leader_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } health_report { overall_health: HEALTHY } } }
I20261118 02:55:05.147773 3340 catalog_manager.cc:5840] T 2c92955aa65142f1bd20f4b8f3321500 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } }
I20261118 02:55:05.147877 3881 tablet_service.cc:1500] Processing CreateTablet for tablet 8bed657d0da64d95851f8a86216b6c2c (DEFAULT_TABLE table=default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:05.148326 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bed657d0da64d95851f8a86216b6c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:05.148185 3340 catalog_manager.cc:5840] T 1904140da78b4567a9decb250c6f74a3 P 99a39b1d9d814deeb3a8a644e9b392ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65). New cstate: current_term: 1 leader_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } }
@ 0x7f72ad19fb23 google::LogMessage::SendToLog() at ??:0
@ 0x7f72ad19b75c google::LogMessage::Flush() at ??:0
@ 0x7f72ad19b9a1 google::LogMessage::~LogMessage() at ??:0
I20261118 02:55:05.153110 3734 tablet_service.cc:1500] Processing CreateTablet for tablet 8bed657d0da64d95851f8a86216b6c2c (DEFAULT_TABLE table=default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:05.153513 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bed657d0da64d95851f8a86216b6c2c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f72c261b9a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 02:55:05.159298 3597 tablet_service.cc:1500] Processing CreateTablet for tablet 8bed657d0da64d95851f8a86216b6c2c (DEFAULT_TABLE table=default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:05.159722 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bed657d0da64d95851f8a86216b6c2c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f72c261a251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20261118 02:55:05.166616 4328 tablet_bootstrap.cc:492] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:05.168407 4328 tablet_bootstrap.cc:654] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
@ 0x7f72c2621ecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20261118 02:55:05.169085 4323 tablet_bootstrap.cc:492] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:05.170626 4341 tablet_bootstrap.cc:492] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:05.170773 4323 tablet_bootstrap.cc:654] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:05.171552 3340 master_service.cc:965] ReplaceTablet: received request to replace tablet 489beea71ae74236addee59c153cfee7 from {username='slave'} at 127.0.0.1:48092
@ 0x7f72c2621d49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
I20261118 02:55:05.173688 4341 tablet_bootstrap.cc:654] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:05.175356 4328 tablet_bootstrap.cc:492] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:05.175529 4328 ts_tablet_manager.cc:1449] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
@ 0x7f72ae5e4dc9 kudu::Thread::SuperviseThread() at ??:0
I20261118 02:55:05.176349 4328 raft_consensus.cc:360] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.176560 4328 raft_consensus.cc:386] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:05.176623 4328 raft_consensus.cc:748] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:05.176872 4328 consensus_queue.cc:261] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.178706 4328 ts_tablet_manager.cc:1480] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:05.179394 3340 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet 489beea71ae74236addee59c153cfee7
I20261118 02:55:05.180143 3595 tablet_service.cc:1547] Processing DeleteTablet for tablet 489beea71ae74236addee59c153cfee7 with delete_type TABLET_DATA_DELETED (replaced by tablet ccf78258c9634540a178855a142f4d4b) from {username='slave'} at 127.0.0.1:53002
I20261118 02:55:05.180500 3340 catalog_manager.cc:6936] ReplaceTablet: tablet 489beea71ae74236addee59c153cfee7 deleted and replaced by tablet ccf78258c9634540a178855a142f4d4b
I20261118 02:55:05.180761 3734 tablet_service.cc:1547] Processing DeleteTablet for tablet 489beea71ae74236addee59c153cfee7 with delete_type TABLET_DATA_DELETED (replaced by tablet ccf78258c9634540a178855a142f4d4b) from {username='slave'} at 127.0.0.1:57892
I20261118 02:55:05.180630 3340 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:48092 (request call id 31) took 1320 ms. Trace:
I20261118 02:55:05.182221 3880 tablet_service.cc:1547] Processing DeleteTablet for tablet 489beea71ae74236addee59c153cfee7 with delete_type TABLET_DATA_DELETED (replaced by tablet ccf78258c9634540a178855a142f4d4b) from {username='slave'} at 127.0.0.1:50250
I20261118 02:55:05.182708 3340 rpcz_store.cc:276] 1118 02:55:03.860229 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 02:55:05.170378 (+1310149us) service_pool.cc:224] Handling call
1118 02:55:05.180578 (+ 10200us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 02:55:05.172773 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 02:55:05.172891 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
1118 02:55:05.172898 (+ 7us) write_op.cc:435] Acquired schema lock
1118 02:55:05.172900 (+ 2us) tablet.cc:670] Decoding operations
1118 02:55:05.173012 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
1118 02:55:05.173026 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1118 02:55:05.173031 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 02:55:05.173107 (+ 76us) tablet.cc:709] Row locks acquired
1118 02:55:05.173109 (+ 2us) write_op.cc:260] PREPARE: finished
1118 02:55:05.173144 (+ 35us) write_op.cc:270] Start()
1118 02:55:05.173176 (+ 32us) write_op.cc:276] Timestamp: P: 1794970505173118 usec, L: 0
1118 02:55:05.173180 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 02:55:05.173350 (+ 170us) log.cc:844] Serialized 658 byte log entry
1118 02:55:05.176965 (+ 3615us) op_driver.cc:466] REPLICATION: finished
1118 02:55:05.177137 (+ 172us) write_op.cc:301] APPLY: starting
1118 02:55:05.177153 (+ 16us) tablet.cc:1402] starting BulkCheckPresence
1118 02:55:05.177203 (+ 50us) tablet.cc:1405] finished BulkCheckPresence
1118 02:55:05.177232 (+ 29us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 02:55:05.177334 (+ 102us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 02:55:05.177383 (+ 49us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1118 02:55:05.177391 (+ 8us) write_op.cc:312] APPLY: finished
1118 02:55:05.177461 (+ 70us) log.cc:844] Serialized 28 byte log entry
1118 02:55:05.177525 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:05.177547 (+ 22us) write_op.cc:454] Released schema lock
1118 02:55:05.177553 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1118 02:55:05.177634 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
1118 02:55:05.177638 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":119,"apply.run_cpu_time_us":488,"apply.run_wall_time_us":488,"num_ops":2,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":702,"prepare.run_wall_time_us":702,"raft.queue_time_us":560,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":302,"replication_time_us":3754}]]}
I20261118 02:55:05.189301 3340 master_service.cc:965] ReplaceTablet: received request to replace tablet a013d180f2cd430ab1ba30b39fd27591 from {username='slave'} at 127.0.0.1:48092
I20261118 02:55:05.189515 4399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:05.189739 4399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:05.189939 3735 tablet_service.cc:1500] Processing CreateTablet for tablet ccf78258c9634540a178855a142f4d4b (DEFAULT_TABLE table=default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:05.190398 3735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf78258c9634540a178855a142f4d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:05.190337 3594 tablet_service.cc:1500] Processing CreateTablet for tablet ccf78258c9634540a178855a142f4d4b (DEFAULT_TABLE table=default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:05.190773 3594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf78258c9634540a178855a142f4d4b. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f72b1cbf6db start_thread at ??:0
@ 0x7f72aa72f61f clone at ??:0
I20261118 02:55:05.200424 4323 tablet_bootstrap.cc:492] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:05.200577 4410 tablet_replica.cc:333] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:05.200605 4323 ts_tablet_manager.cc:1449] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20261118 02:55:05.200768 4410 raft_consensus.cc:2261] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:05.201126 4410 raft_consensus.cc:2290] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:05.201323 4323 raft_consensus.cc:360] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.201519 4323 raft_consensus.cc:386] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:05.201596 4323 raft_consensus.cc:748] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:05.201702 4410 ts_tablet_manager.cc:1962] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:05.201846 4323 consensus_queue.cc:261] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.202301 4323 ts_tablet_manager.cc:1480] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:05.202649 4323 tablet_bootstrap.cc:492] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:05.206444 4323 tablet_bootstrap.cc:654] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:05.207084 4408 tablet_replica.cc:333] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:05.207288 4408 raft_consensus.cc:2261] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:05.207480 4408 raft_consensus.cc:2290] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:05.208034 4408 ts_tablet_manager.cc:1962] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:05.211724 4323 tablet_bootstrap.cc:492] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:05.211874 4323 ts_tablet_manager.cc:1449] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20261118 02:55:05.212636 4409 tablet_replica.cc:333] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:05.212711 4323 raft_consensus.cc:360] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:05.212854 4409 raft_consensus.cc:2261] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:05.212908 4323 raft_consensus.cc:386] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:05.212996 4323 raft_consensus.cc:748] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:05.213106 4409 raft_consensus.cc:2290] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:05.213279 4323 consensus_queue.cc:261] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:05.213687 4409 ts_tablet_manager.cc:1962] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:05.215392 4323 ts_tablet_manager.cc:1480] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:05.216957 4341 tablet_bootstrap.cc:492] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:05.217123 4341 ts_tablet_manager.cc:1449] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.015s
I20261118 02:55:05.217851 4341 raft_consensus.cc:360] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.218035 4341 raft_consensus.cc:386] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:05.218169 4341 raft_consensus.cc:748] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:05.218425 4341 consensus_queue.cc:261] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.218950 4341 ts_tablet_manager.cc:1480] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 02:55:05.220050 4341 tablet_bootstrap.cc:492] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:05.221827 4341 tablet_bootstrap.cc:654] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:05.223308 4410 ts_tablet_manager.cc:1975] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:05.223424 4410 log.cc:1199] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/489beea71ae74236addee59c153cfee7
I20261118 02:55:05.223821 4410 ts_tablet_manager.cc:1996] T 489beea71ae74236addee59c153cfee7 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:05.227204 4408 ts_tablet_manager.cc:1975] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:05.227327 4408 log.cc:1199] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/489beea71ae74236addee59c153cfee7
I20261118 02:55:05.227715 4408 ts_tablet_manager.cc:1996] T 489beea71ae74236addee59c153cfee7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:05.228267 4341 tablet_bootstrap.cc:492] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:05.228411 4341 ts_tablet_manager.cc:1449] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20261118 02:55:05.229288 4341 raft_consensus.cc:360] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:05.229461 4341 raft_consensus.cc:386] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:05.229548 4341 raft_consensus.cc:748] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:05.229779 4341 consensus_queue.cc:261] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:05.230417 4341 ts_tablet_manager.cc:1480] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:05.230805 4409 ts_tablet_manager.cc:1975] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:05.230931 4409 log.cc:1199] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/489beea71ae74236addee59c153cfee7
I20261118 02:55:05.231369 4409 ts_tablet_manager.cc:1996] T 489beea71ae74236addee59c153cfee7 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:05.262112 4379 raft_consensus.cc:499] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:05.262317 4379 raft_consensus.cc:521] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.263139 4379 leader_election.cc:290] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:05.263607 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8bed657d0da64d95851f8a86216b6c2c" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:05.263814 3901 raft_consensus.cc:2490] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:05.263940 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8bed657d0da64d95851f8a86216b6c2c" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:05.264169 3616 raft_consensus.cc:2490] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:05.264343 3690 leader_election.cc:304] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:05.264626 4379 raft_consensus.cc:2826] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:05.264745 4379 raft_consensus.cc:499] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:05.264823 4379 raft_consensus.cc:3096] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:05.266711 4379 raft_consensus.cc:521] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.267699 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8bed657d0da64d95851f8a86216b6c2c" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:05.267707 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8bed657d0da64d95851f8a86216b6c2c" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:05.267884 3616 raft_consensus.cc:3096] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:05.267896 3901 raft_consensus.cc:3096] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:05.270753 3901 raft_consensus.cc:2490] T 8bed657d0da64d95851f8a86216b6c2c P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:05.270768 3616 raft_consensus.cc:2490] T 8bed657d0da64d95851f8a86216b6c2c P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:05.271226 3691 leader_election.cc:304] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:05.271725 4379 leader_election.cc:290] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:05.271934 4379 raft_consensus.cc:2826] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:05.272045 4379 raft_consensus.cc:705] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:05.272369 4379 consensus_queue.cc:238] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.285882 4421 consensus_queue.cc:1049] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:05.288618 4423 consensus_queue.cc:1049] T 8bed657d0da64d95851f8a86216b6c2c P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 02:55:05.426714 3691 connection.cc:573] server connection from 127.0.0.1:57892 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:05.426754 3553 connection.cc:573] server connection from 127.0.0.1:53002 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:05.426754 3835 connection.cc:573] server connection from 127.0.0.1:50250 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:05.427068 4137 connection.cc:573] server connection from 127.0.0.1:34230 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:05.427281 3836 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:05.427372 3689 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:05.428056 3552 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:05.428491 3955 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:05.428947 3955 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.229.125:38339)
W20261118 02:55:05.430042 3955 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.229.125:38339)
I20261118 02:55:05.642093 4421 raft_consensus.cc:499] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:05.642302 4421 raft_consensus.cc:521] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.642930 4421 leader_election.cc:290] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:05.643451 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "0e901a03b90e4c978eb98be87163cfe5" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:05.643455 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "0e901a03b90e4c978eb98be87163cfe5" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:05.643636 3616 raft_consensus.cc:2490] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:05.643646 3901 raft_consensus.cc:2490] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:05.644119 3691 leader_election.cc:304] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:05.644411 4421 raft_consensus.cc:2826] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:05.644507 4421 raft_consensus.cc:499] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:05.644608 4421 raft_consensus.cc:3096] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:05.646481 4421 raft_consensus.cc:521] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.647061 4421 leader_election.cc:290] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:05.647534 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "0e901a03b90e4c978eb98be87163cfe5" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:05.647552 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "0e901a03b90e4c978eb98be87163cfe5" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:05.647730 3616 raft_consensus.cc:3096] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:05.647730 3901 raft_consensus.cc:3096] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:05.650602 3616 raft_consensus.cc:2490] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:05.650617 3901 raft_consensus.cc:2490] T 0e901a03b90e4c978eb98be87163cfe5 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:05.651042 3691 leader_election.cc:304] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:05.651304 4421 raft_consensus.cc:2826] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:05.651455 4421 raft_consensus.cc:705] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:05.651746 4421 consensus_queue.cc:238] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:05.654352 4423 consensus_queue.cc:1049] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:05.654570 4379 consensus_queue.cc:1049] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:05.698751 4437 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:05.698905 4437 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:05.699532 4437 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:05.700556 4139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:05.701692 4138 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:05.701833 4139 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:05.701985 4139 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:05.702286 4437 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261118 02:55:05.776943 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.126:38025
--webserver_interface=127.2.229.126
--webserver_port=44955
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 02:55:05.836552 4421 raft_consensus.cc:499] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:05.836755 4421 raft_consensus.cc:521] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:05.837512 4421 leader_election.cc:290] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:05.838105 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ccf78258c9634540a178855a142f4d4b" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:05.838363 3616 raft_consensus.cc:2490] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
W20261118 02:55:05.838675 3690 leader_election.cc:343] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: ccf78258c9634540a178855a142f4d4b
I20261118 02:55:05.838943 3691 leader_election.cc:304] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 3f1973986ec1466eab5bec01dd81654f
I20261118 02:55:05.839244 4421 raft_consensus.cc:2826] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:05.839365 4421 raft_consensus.cc:499] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:05.839455 4421 raft_consensus.cc:3096] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:05.842037 4421 raft_consensus.cc:521] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:05.842864 4421 leader_election.cc:290] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:05.843354 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ccf78258c9634540a178855a142f4d4b" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:05.843525 3616 raft_consensus.cc:3096] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
W20261118 02:55:05.843816 3690 leader_election.cc:343] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: ccf78258c9634540a178855a142f4d4b
I20261118 02:55:05.845495 3616 raft_consensus.cc:2490] T ccf78258c9634540a178855a142f4d4b P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:05.845882 3691 leader_election.cc:304] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 3f1973986ec1466eab5bec01dd81654f
I20261118 02:55:05.846185 4421 raft_consensus.cc:2826] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:05.846311 4421 raft_consensus.cc:705] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:05.846582 4421 consensus_queue.cc:238] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
W20261118 02:55:05.850759 3690 consensus_peers.cc:597] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Couldn't send request to peer 3f1973986ec1466eab5bec01dd81654f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccf78258c9634540a178855a142f4d4b. This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:05.850984 4421 consensus_queue.cc:1049] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 02:55:06.139664 4441 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:55:06.139981 4441 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:55:06.140116 4441 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:55:06.148783 4441 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:55:06.148885 4441 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:55:06.148962 4441 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:55:06.149024 4441 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:55:06.159363 4441 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.126:38025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.229.126
--webserver_port=44955
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:55:06.161120 4441 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:55:06.163302 4441 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:55:06.172415 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:06.172385 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:06.172384 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:55:06.173213 4441 server_base.cc:1061] running on GCE node
I20261118 02:55:06.173975 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:55:06.175623 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:55:06.176844 4441 hybrid_clock.cc:648] HybridClock initialized: now 1794970506176791 us; error 45 us; skew 500 ppm
I20261118 02:55:06.177234 4441 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:55:06.179368 4441 webserver.cc:577] Webserver started at http://127.2.229.126:44955/ using document root <none> and password file <none>
I20261118 02:55:06.180065 4441 fs_manager.cc:362] Metadata directory not provided
I20261118 02:55:06.180224 4441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:55:06.185964 4441 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20261118 02:55:06.189163 4457 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:55:06.190706 4441 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20261118 02:55:06.190864 4441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
uuid: "26c6b20fa203404a88463ea27bdd3c7d"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:55:06.191581 4441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:55:06.218504 4441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:55:06.219444 4441 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:55:06.219714 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:55:06.238757 4441 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.126:38025
I20261118 02:55:06.238776 4508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.126:38025 every 8 connection(s)
I20261118 02:55:06.240468 4441 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/info.pb
I20261118 02:55:06.246315 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 4441
I20261118 02:55:06.248207 4509 sys_catalog.cc:263] Verifying existing consensus state
I20261118 02:55:06.252027 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Bootstrap starting.
I20261118 02:55:06.270102 4509 log.cc:826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:06.312275 4513 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:06.312477 4513 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:06.313138 4513 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:06.321758 4138 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:06.339010 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=85 overwritten=0 applied=84 ignored=0} inserts{seen=56 ignored=0} mutations{seen=70 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20261118 02:55:06.339728 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Bootstrap complete.
I20261118 02:55:06.347797 4509 raft_consensus.cc:360] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:06.351295 4509 raft_consensus.cc:748] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c6b20fa203404a88463ea27bdd3c7d, State: Initialized, Role: FOLLOWER
I20261118 02:55:06.352030 4509 consensus_queue.cc:261] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 4.85, Last appended by leader: 85, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:06.352927 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:06.353240 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:06.354193 4509 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:55:06.355332 4484 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 93 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:55:06.355957 4484 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 813a6937cea442ef81645fccc32207f4 in term 4.
I20261118 02:55:06.356923 4139 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4; no voters: 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:06.357728 4513 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20261118 02:55:06.357846 4513 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:06.357932 4513 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 4 FOLLOWER]: Advancing to term 5
I20261118 02:55:06.361764 4513 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:06.366215 4513 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 election: Requested vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:06.366958 4138 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:06.368651 4484 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 5 candidate_status { last_received { term: 4 index: 93 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d"
I20261118 02:55:06.368886 4484 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 4 FOLLOWER]: Advancing to term 5
I20261118 02:55:06.370718 3800 heartbeater.cc:345] Connected to a master server at 127.2.229.126:38025
I20261118 02:55:06.375556 3663 heartbeater.cc:345] Connected to a master server at 127.2.229.126:38025
I20261118 02:55:06.381558 4484 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 813a6937cea442ef81645fccc32207f4 in term 5.
I20261118 02:55:06.382061 4139 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4; no voters: 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:06.382437 4513 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 5 FOLLOWER]: Leader election won for term 5
I20261118 02:55:06.383000 4513 raft_consensus.cc:705] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 5 LEADER]: Becoming Leader. State: Replica: 813a6937cea442ef81645fccc32207f4, State: Running, Role: LEADER
I20261118 02:55:06.383437 4513 consensus_queue.cc:238] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 4.93, Last appended by leader: 93, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:06.385298 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 813a6937cea442ef81645fccc32207f4. Latest consensus state: current_term: 5 leader_uuid: "813a6937cea442ef81645fccc32207f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:06.385601 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:06.392979 4536 catalog_manager.cc:820] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20261118 02:55:06.396342 4138 consensus_peers.cc:597] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 -> Peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Couldn't send request to peer 457eb38cba564ecd9f6f11245172260b. Status: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:06.398357 4539 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 02:55:06.402362 4539 catalog_manager.cc:694] Loaded metadata for table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190]
I20261118 02:55:06.402966 4539 catalog_manager.cc:694] Loaded metadata for table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c]
I20261118 02:55:06.403434 4539 catalog_manager.cc:694] Loaded metadata for table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]
I20261118 02:55:06.403913 4539 catalog_manager.cc:694] Loaded metadata for table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]
I20261118 02:55:06.404709 4539 catalog_manager.cc:694] Loaded metadata for table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]
I20261118 02:55:06.405172 4539 catalog_manager.cc:694] Loaded metadata for table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]
I20261118 02:55:06.405908 4539 catalog_manager.cc:694] Loaded metadata for table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da]
I20261118 02:55:06.406381 4539 catalog_manager.cc:694] Loaded metadata for table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]
I20261118 02:55:06.406839 4539 catalog_manager.cc:694] Loaded metadata for table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d]
I20261118 02:55:06.407336 4539 catalog_manager.cc:694] Loaded metadata for table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]
I20261118 02:55:06.407795 4539 catalog_manager.cc:694] Loaded metadata for table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]
I20261118 02:55:06.408233 4539 catalog_manager.cc:694] Loaded metadata for table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b]
I20261118 02:55:06.408658 4539 catalog_manager.cc:694] Loaded metadata for table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]
I20261118 02:55:06.409092 4539 catalog_manager.cc:694] Loaded metadata for table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]
I20261118 02:55:06.409512 4539 catalog_manager.cc:694] Loaded metadata for table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]
I20261118 02:55:06.409970 4539 catalog_manager.cc:694] Loaded metadata for table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]
I20261118 02:55:06.410444 4539 catalog_manager.cc:694] Loaded metadata for table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]
I20261118 02:55:06.410789 4484 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Refusing update from remote peer 813a6937cea442ef81645fccc32207f4: Log matching property violated. Preceding OpId in replica: term: 4 index: 85. Preceding OpId from leader: term: 5 index: 94. (index mismatch)
I20261118 02:55:06.410887 4539 catalog_manager.cc:694] Loaded metadata for table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1]
I20261118 02:55:06.411294 4539 catalog_manager.cc:694] Loaded metadata for table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f]
I20261118 02:55:06.411914 4539 catalog_manager.cc:694] Loaded metadata for table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]
I20261118 02:55:06.411827 4541 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 84, Time since last communication: 0.000s
I20261118 02:55:06.414538 4539 tablet_loader.cc:96] loaded metadata for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7])
I20261118 02:55:06.415033 4539 tablet_loader.cc:96] loaded metadata for tablet 0e901a03b90e4c978eb98be87163cfe5 (table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190])
I20261118 02:55:06.415480 4539 tablet_loader.cc:96] loaded metadata for tablet 1904140da78b4567a9decb250c6f74a3 (table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f])
I20261118 02:55:06.415928 4539 tablet_loader.cc:96] loaded metadata for tablet 1d347c0ae53241708ec623562f4740d4 (table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b])
I20261118 02:55:06.416352 4539 tablet_loader.cc:96] loaded metadata for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db])
I20261118 02:55:06.416790 4539 tablet_loader.cc:96] loaded metadata for tablet 2c92955aa65142f1bd20f4b8f3321500 (table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d])
I20261118 02:55:06.418406 4539 tablet_loader.cc:96] loaded metadata for tablet 5f05a8e5b21945a5bd3df86a5467a615 (table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf])
I20261118 02:55:06.418672 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 813a6937cea442ef81645fccc32207f4. Latest consensus state: current_term: 5 leader_uuid: "813a6937cea442ef81645fccc32207f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:06.418823 4539 tablet_loader.cc:96] loaded metadata for tablet 65bd6fcaed5b47c3a2478d375ace9636 (table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73])
I20261118 02:55:06.418919 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:06.419395 4539 tablet_loader.cc:96] loaded metadata for tablet 72f427c72a554ac1a0a2f2c3a78865da (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b])
I20261118 02:55:06.419799 4539 tablet_loader.cc:96] loaded metadata for tablet 78189cccd457466bb5d01231038fec24 (table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da])
I20261118 02:55:06.420197 4539 tablet_loader.cc:96] loaded metadata for tablet 8bed657d0da64d95851f8a86216b6c2c (table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c])
I20261118 02:55:06.420630 4539 tablet_loader.cc:96] loaded metadata for tablet 8f853e6b923444efa79cfbab2e31540e (table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec])
I20261118 02:55:06.421502 4539 tablet_loader.cc:96] loaded metadata for tablet a013d180f2cd430ab1ba30b39fd27591 (table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d])
I20261118 02:55:06.422183 4539 tablet_loader.cc:96] loaded metadata for tablet b129518ce23c45758ebcaa55410757d8 (table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1])
I20261118 02:55:06.423887 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "813a6937cea442ef81645fccc32207f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:06.424127 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:06.428741 4539 tablet_loader.cc:96] loaded metadata for tablet b610080fc91745c7be260699946d5681 (table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139])
I20261118 02:55:06.429382 4539 tablet_loader.cc:96] loaded metadata for tablet c92c29c30ab6411dabe6f02331d29b7a (table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d])
I20261118 02:55:06.430320 4539 tablet_loader.cc:96] loaded metadata for tablet ccf78258c9634540a178855a142f4d4b (table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e])
I20261118 02:55:06.426038 4441 master_runner.cc:187] Error getting master registration for 127.2.229.125:38339: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111),
I20261118 02:55:06.430853 4441 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:06.430922 4539 tablet_loader.cc:96] loaded metadata for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6])
I20261118 02:55:06.432561 4539 tablet_loader.cc:96] loaded metadata for tablet f094af4ac3604326a539461d06b01868 (table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67])
I20261118 02:55:06.433629 4539 tablet_loader.cc:96] loaded metadata for tablet ffe8d4c111d14054b76f9c7724072a94 (table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66])
I20261118 02:55:06.434425 4539 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 02:55:06.435967 4539 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:06.436036 4539 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:55:06.437469 4539 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:55:06.438617 4539 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Loaded TSK: 0
I20261118 02:55:06.439740 4539 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 02:55:06.439996 4513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "813a6937cea442ef81645fccc32207f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:06.440294 4513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:06.440943 4539 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:55:06.441959 4539 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: 20
I20261118 02:55:06.488780 4150 master_service.cc:965] ReplaceTablet: received request to replace tablet f094af4ac3604326a539461d06b01868 from {username='slave'} at 127.0.0.1:34238
I20261118 02:55:06.489176 4145 master_service.cc:965] ReplaceTablet: received request to replace tablet 8f853e6b923444efa79cfbab2e31540e from {username='slave'} at 127.0.0.1:34238
I20261118 02:55:06.487730 4149 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34238:
name: "default.Table_51d36ff687ca47b5995e5693d7b8931b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:06.487991 4152 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34238:
name: "default.Table_1d3bb660e14d4101996960be8caa8f81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:06.487829 4146 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34238:
table { table_name: "default.Table_287099a68a3540f1be0284c71d4d7056" } modify_external_catalogs: true
I20261118 02:55:06.488462 4148 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34238:
name: "default.Table_ee3815d9524243ec886bd171ccaff9e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:06.493001 4146 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34238:
table { table_name: "default.Table_287099a68a3540f1be0284c71d4d7056" } modify_external_catalogs: true
I20261118 02:55:06.490669 4153 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34238:
name: "default.Table_52fca7a2e65f4e679bbef9de3df569f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:06.490001 4151 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34238:
name: "default.Table_8139578347f041afbe51671d3be78447"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:06.488179 4147 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34238:
table { table_name: "default.Table_571d1360811c42dc815efb439b2b52e6" } modify_external_catalogs: true
I20261118 02:55:06.488986 4144 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34238:
name: "default.Table_40868189cecf45d0b705461c4328ee17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:06.493492 4147 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34238:
table { table_name: "default.Table_571d1360811c42dc815efb439b2b52e6" } modify_external_catalogs: true
F20261118 02:55:06.502859 4205 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 21, previous 21; dumping first 2 out of 15 received events: NotificationEvent(eventId=21, eventTime=1794970503, eventType=CREATE_TABLE, dbName=default, tableName=table_f0ae490a0df64e1097e144c7a370df61, message={"server":"","servicePrincipal":"","db":"default","table":"table_f0ae490a0df64e1097e144c7a370df61","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f0ae490a0df64e1097e144c7a370df61\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970503},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_f0ae490a0df64e1097e144c7a370df61\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f0ae490a0df64e1097e144c7a370df61\",\"kudu.table_id\":\"a59b9425c01049269883cf7767936c67\",\"transient_lastDdlTime\":\"1794970503\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1794970503, eventType=DROP_TABLE, dbName=default, tableName=table_c113c2b57e17479e92e241234a241af7, message={"server":"","servicePrincipal":"","db":"default","table":"table_c113c2b57e17479e92e241234a241af7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c113c2b57e17479e92e241234a241af7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970500},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_c113c2b57e17479e92e241234a241af7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c113c2b57e17479e92e241234a241af7\",\"kudu.table_id\":\"da0cb25ccfed426ca61f0dd9f93d30ec\",\"transient_lastDdlTime\":\"1794970500\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7ff7582a7d5d google::LogMessage::Fail() at ??:0
@ 0x7ff7582abb23 google::LogMessage::SendToLog() at ??:0
@ 0x7ff7582a775c google::LogMessage::Flush() at ??:0
@ 0x7ff7582a79a1 google::LogMessage::~LogMessage() at ??:0
I20261118 02:55:06.517438 4145 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Sending DeleteTablet for 3 replicas of tablet 8f853e6b923444efa79cfbab2e31540e
I20261118 02:55:06.524168 4150 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Sending DeleteTablet for 3 replicas of tablet f094af4ac3604326a539461d06b01868
I20261118 02:55:06.524173 4145 catalog_manager.cc:6936] ReplaceTablet: tablet 8f853e6b923444efa79cfbab2e31540e deleted and replaced by tablet c0713dcf79f54873a99d7556ad8f1454
I20261118 02:55:06.529067 4145 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34238:
name: "default.Table_bdbed231860946c2bf9b514f724bc3e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:06.530020 4150 catalog_manager.cc:6936] ReplaceTablet: tablet f094af4ac3604326a539461d06b01868 deleted and replaced by tablet ab61b43e639a4e75bc2e7009ea86a64f
I20261118 02:55:06.530406 4150 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34238:
table { table_name: "default.Table_87d592a09c3348028719a603956fbce6" } modify_external_catalogs: true
I20261118 02:55:06.530674 4150 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34238:
table { table_name: "default.Table_87d592a09c3348028719a603956fbce6" } modify_external_catalogs: true
@ 0x7ff76d7279a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 02:55:06.538004 3595 tablet_service.cc:1547] Processing DeleteTablet for tablet 8f853e6b923444efa79cfbab2e31540e with delete_type TABLET_DATA_DELETED (replaced by tablet c0713dcf79f54873a99d7556ad8f1454) from {username='slave'} at 127.0.0.1:33682
I20261118 02:55:06.538363 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet f094af4ac3604326a539461d06b01868 with delete_type TABLET_DATA_DELETED (replaced by tablet ab61b43e639a4e75bc2e7009ea86a64f) from {username='slave'} at 127.0.0.1:33682
I20261118 02:55:06.554519 4558 tablet_replica.cc:333] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:06.554770 4558 raft_consensus.cc:2261] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:06.555022 4558 raft_consensus.cc:2290] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:06.555778 4558 ts_tablet_manager.cc:1962] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:06.559333 3595 tablet_service.cc:1500] Processing CreateTablet for tablet c0713dcf79f54873a99d7556ad8f1454 (DEFAULT_TABLE table=default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:06.559973 3595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0713dcf79f54873a99d7556ad8f1454. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:06.560336 3597 tablet_service.cc:1500] Processing CreateTablet for tablet ab61b43e639a4e75bc2e7009ea86a64f (DEFAULT_TABLE table=default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:06.560724 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab61b43e639a4e75bc2e7009ea86a64f. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7ff76d726251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7ff76d72decc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7ff76d72dd49 std::_Function_handler<>::_M_invoke() at ??:0
I20261118 02:55:06.576813 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet f094af4ac3604326a539461d06b01868 with delete_type TABLET_DATA_DELETED (replaced by tablet ab61b43e639a4e75bc2e7009ea86a64f) from {username='slave'} at 127.0.0.1:56632
I20261118 02:55:06.576848 3734 tablet_service.cc:1547] Processing DeleteTablet for tablet 8f853e6b923444efa79cfbab2e31540e with delete_type TABLET_DATA_DELETED (replaced by tablet c0713dcf79f54873a99d7556ad8f1454) from {username='slave'} at 127.0.0.1:48148
I20261118 02:55:06.577010 3880 tablet_service.cc:1547] Processing DeleteTablet for tablet 8f853e6b923444efa79cfbab2e31540e with delete_type TABLET_DATA_DELETED (replaced by tablet c0713dcf79f54873a99d7556ad8f1454) from {username='slave'} at 127.0.0.1:56632
I20261118 02:55:06.577426 4558 ts_tablet_manager.cc:1975] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:06.577543 4558 log.cc:1199] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/8f853e6b923444efa79cfbab2e31540e
I20261118 02:55:06.577507 3879 tablet_service.cc:1500] Processing CreateTablet for tablet c0713dcf79f54873a99d7556ad8f1454 (DEFAULT_TABLE table=default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:06.577483 3882 tablet_service.cc:1500] Processing CreateTablet for tablet ab61b43e639a4e75bc2e7009ea86a64f (DEFAULT_TABLE table=default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:06.577947 3879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0713dcf79f54873a99d7556ad8f1454. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:06.577948 4558 ts_tablet_manager.cc:1996] T 8f853e6b923444efa79cfbab2e31540e P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:06.578984 4558 tablet_replica.cc:333] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:06.579005 4140 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet 8f853e6b923444efa79cfbab2e31540e (table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]) successfully deleted
I20261118 02:55:06.579177 4558 raft_consensus.cc:2261] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:06.579331 4558 raft_consensus.cc:2290] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:06.579869 4558 ts_tablet_manager.cc:1962] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:06.581113 3735 tablet_service.cc:1547] Processing DeleteTablet for tablet f094af4ac3604326a539461d06b01868 with delete_type TABLET_DATA_DELETED (replaced by tablet ab61b43e639a4e75bc2e7009ea86a64f) from {username='slave'} at 127.0.0.1:48148
I20261118 02:55:06.581939 3735 tablet_service.cc:1500] Processing CreateTablet for tablet c0713dcf79f54873a99d7556ad8f1454 (DEFAULT_TABLE table=default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:06.582327 3735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0713dcf79f54873a99d7556ad8f1454. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:06.584251 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab61b43e639a4e75bc2e7009ea86a64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:06.585860 4558 ts_tablet_manager.cc:1975] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:06.585973 4558 log.cc:1199] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/f094af4ac3604326a539461d06b01868
I20261118 02:55:06.586333 4558 ts_tablet_manager.cc:1996] T f094af4ac3604326a539461d06b01868 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:06.587374 4140 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet f094af4ac3604326a539461d06b01868 (table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]) successfully deleted
I20261118 02:55:06.587672 4563 tablet_bootstrap.cc:492] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:06.589735 4563 tablet_bootstrap.cc:654] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:06.590438 3733 tablet_service.cc:1500] Processing CreateTablet for tablet ab61b43e639a4e75bc2e7009ea86a64f (DEFAULT_TABLE table=default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:06.590842 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab61b43e639a4e75bc2e7009ea86a64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:06.591141 4559 tablet_bootstrap.cc:492] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:06.592049 4563 tablet_bootstrap.cc:492] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:06.592204 4563 ts_tablet_manager.cc:1449] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
I20261118 02:55:06.592983 4559 tablet_bootstrap.cc:654] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:06.592933 4563 raft_consensus.cc:360] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.593106 4563 raft_consensus.cc:386] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:06.593182 4563 raft_consensus.cc:748] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:06.593428 4563 consensus_queue.cc:261] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.603297 3947 heartbeater.cc:345] Connected to a master server at 127.2.229.126:38025
I20261118 02:55:06.603652 4563 ts_tablet_manager.cc:1480] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20261118 02:55:06.603997 4563 tablet_bootstrap.cc:492] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
@ 0x7ff7596f0dc9 kudu::Thread::SuperviseThread() at ??:0
I20261118 02:55:06.606629 4474 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" instance_seqno: 1794970499527172) as {username='slave'} at 127.2.229.67:39447; Asking this server to re-register.
@ 0x7ff75cdcb6db start_thread at ??:0
I20261118 02:55:06.607681 3947 heartbeater.cc:462] Registering TS with master...
@ 0x7ff75583b61f clone at ??:0
I20261118 02:55:06.609266 4564 tablet_bootstrap.cc:492] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:06.610337 4474 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:06.611084 4559 tablet_bootstrap.cc:492] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:06.611208 4564 tablet_bootstrap.cc:654] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:06.611228 4559 ts_tablet_manager.cc:1449] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20261118 02:55:06.611785 4563 tablet_bootstrap.cc:654] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:06.613248 4559 raft_consensus.cc:360] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.613437 4559 raft_consensus.cc:386] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:06.613516 4559 raft_consensus.cc:748] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:06.614686 4559 consensus_queue.cc:261] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.615836 4561 tablet_replica.cc:333] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:06.616035 4561 raft_consensus.cc:2261] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:06.616442 4561 raft_consensus.cc:2290] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:06.617007 4561 ts_tablet_manager.cc:1962] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:06.618352 4559 ts_tablet_manager.cc:1480] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20261118 02:55:06.619112 4474 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" instance_seqno: 1794970498532688) as {username='slave'} at 127.2.229.65:52805; Asking this server to re-register.
I20261118 02:55:06.620012 3663 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:06.622020 4474 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:06.622129 4559 tablet_bootstrap.cc:492] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:06.623769 4563 tablet_bootstrap.cc:492] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:06.623953 4563 ts_tablet_manager.cc:1449] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.001s
I20261118 02:55:06.624855 4563 raft_consensus.cc:360] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.625077 4563 raft_consensus.cc:386] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:06.625144 4563 raft_consensus.cc:748] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:06.625272 4562 tablet_replica.cc:333] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:06.625474 4562 raft_consensus.cc:2261] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:06.625420 4563 consensus_queue.cc:261] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.625664 4562 raft_consensus.cc:2290] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:06.626094 4563 ts_tablet_manager.cc:1480] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261118 02:55:06.626240 4562 ts_tablet_manager.cc:1962] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:06.626706 4559 tablet_bootstrap.cc:654] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:06.632683 4564 tablet_bootstrap.cc:492] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:06.632843 4564 ts_tablet_manager.cc:1449] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20261118 02:55:06.633646 4564 raft_consensus.cc:360] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.633908 4564 raft_consensus.cc:386] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:06.634002 4564 raft_consensus.cc:748] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:06.634089 4562 ts_tablet_manager.cc:1975] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:06.634209 4562 log.cc:1199] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/8f853e6b923444efa79cfbab2e31540e
I20261118 02:55:06.634241 4564 consensus_queue.cc:261] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.634860 4564 ts_tablet_manager.cc:1480] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 02:55:06.640374 4561 ts_tablet_manager.cc:1975] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:06.640482 4561 log.cc:1199] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/f094af4ac3604326a539461d06b01868
I20261118 02:55:06.640900 4562 ts_tablet_manager.cc:1996] T 8f853e6b923444efa79cfbab2e31540e P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:06.640919 4561 ts_tablet_manager.cc:1996] T f094af4ac3604326a539461d06b01868 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:06.642283 4561 tablet_replica.cc:333] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:06.642499 4561 raft_consensus.cc:2261] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:06.642822 4561 raft_consensus.cc:2290] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:06.643400 4561 ts_tablet_manager.cc:1962] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:06.643483 4474 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" instance_seqno: 1794970499028671) as {username='slave'} at 127.2.229.66:40409; Asking this server to re-register.
I20261118 02:55:06.644404 3800 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:06.646291 4559 tablet_bootstrap.cc:492] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:06.646449 4559 ts_tablet_manager.cc:1449] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.004s
I20261118 02:55:06.646463 4474 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:06.647239 4559 raft_consensus.cc:360] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.647430 4559 raft_consensus.cc:386] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:06.647502 4559 raft_consensus.cc:748] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:06.647732 4559 consensus_queue.cc:261] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.648499 4559 ts_tablet_manager.cc:1480] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261118 02:55:06.648844 4564 tablet_bootstrap.cc:492] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:06.650580 4564 tablet_bootstrap.cc:654] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:06.651582 4562 tablet_replica.cc:333] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:06.651782 4562 raft_consensus.cc:2261] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:06.651947 4562 raft_consensus.cc:2290] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:06.652035 4561 ts_tablet_manager.cc:1975] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:06.652185 4561 log.cc:1199] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/8f853e6b923444efa79cfbab2e31540e
I20261118 02:55:06.652527 4561 ts_tablet_manager.cc:1996] T 8f853e6b923444efa79cfbab2e31540e P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:06.652566 4562 ts_tablet_manager.cc:1962] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:06.654943 4564 tablet_bootstrap.cc:492] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:06.655103 4564 ts_tablet_manager.cc:1449] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20261118 02:55:06.655970 4564 raft_consensus.cc:360] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.656200 4564 raft_consensus.cc:386] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:06.656279 4564 raft_consensus.cc:748] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:06.656518 4564 consensus_queue.cc:261] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.657032 4564 ts_tablet_manager.cc:1480] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:06.661882 4562 ts_tablet_manager.cc:1975] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:06.661980 4562 log.cc:1199] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/f094af4ac3604326a539461d06b01868
I20261118 02:55:06.662240 4562 ts_tablet_manager.cc:1996] T f094af4ac3604326a539461d06b01868 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
W20261118 02:55:06.801687 3837 connection.cc:573] server connection from 127.0.0.1:56632 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:06.801677 3689 connection.cc:573] server connection from 127.0.0.1:48148 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:06.801824 3956 connection.cc:573] client connection to 127.2.229.124:36695 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:06.802488 3956 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.229.124:36695)
W20261118 02:55:06.802471 3690 connection.cc:573] client connection to 127.2.229.124:36695 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:06.802870 3956 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.229.124:36695)
W20261118 02:55:06.802850 3553 connection.cc:573] client connection to 127.2.229.124:36695 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:06.803037 3956 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.229.124:36695)
W20261118 02:55:06.803194 3956 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.229.124:36695)
I20261118 02:55:06.925024 4384 raft_consensus.cc:499] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:06.925217 4384 raft_consensus.cc:521] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.925881 4384 leader_election.cc:290] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:06.926492 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c0713dcf79f54873a99d7556ad8f1454" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:06.926473 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c0713dcf79f54873a99d7556ad8f1454" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:06.926703 3751 raft_consensus.cc:2490] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:06.926741 3901 raft_consensus.cc:2490] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:06.927234 3553 leader_election.cc:304] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:06.927500 4384 raft_consensus.cc:2826] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:06.927603 4384 raft_consensus.cc:499] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:06.927668 4384 raft_consensus.cc:3096] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:06.929574 4384 raft_consensus.cc:521] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.930083 4384 leader_election.cc:290] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:06.930624 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c0713dcf79f54873a99d7556ad8f1454" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:06.930621 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c0713dcf79f54873a99d7556ad8f1454" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:06.930785 3751 raft_consensus.cc:3096] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:06.931218 3901 raft_consensus.cc:3096] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:06.933236 3751 raft_consensus.cc:2490] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:06.933617 3552 leader_election.cc:304] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:06.933676 3901 raft_consensus.cc:2490] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:06.933874 4384 raft_consensus.cc:2826] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:06.934281 4384 raft_consensus.cc:705] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:06.934760 4384 consensus_queue.cc:238] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:06.937630 4583 consensus_queue.cc:1049] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:06.938017 4583 consensus_queue.cc:1049] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:06.974995 4591 ts_tablet_manager.cc:979] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: Initiating tablet copy from peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:06.976112 4591 tablet_copy_client.cc:323] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.229.66:40377
I20261118 02:55:06.977246 3775 tablet_copy_service.cc:140] P 99a39b1d9d814deeb3a8a644e9b392ef: Received BeginTabletCopySession request for tablet ccf78258c9634540a178855a142f4d4b from peer 3f1973986ec1466eab5bec01dd81654f ({username='slave'} at 127.2.229.67:42005)
I20261118 02:55:06.977474 3775 tablet_copy_service.cc:161] P 99a39b1d9d814deeb3a8a644e9b392ef: Beginning new tablet copy session on tablet ccf78258c9634540a178855a142f4d4b from peer 3f1973986ec1466eab5bec01dd81654f at {username='slave'} at 127.2.229.67:42005: session id = 3f1973986ec1466eab5bec01dd81654f-ccf78258c9634540a178855a142f4d4b
I20261118 02:55:06.980862 3775 tablet_copy_source_session.cc:215] T ccf78258c9634540a178855a142f4d4b P 99a39b1d9d814deeb3a8a644e9b392ef: Tablet Copy: opened 0 blocks and 1 log segments
I20261118 02:55:06.982932 4591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf78258c9634540a178855a142f4d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:06.987776 4591 tablet_copy_client.cc:806] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: tablet copy: Starting download of 0 data blocks...
I20261118 02:55:06.988040 4591 tablet_copy_client.cc:670] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: tablet copy: Starting download of 1 WAL segments...
I20261118 02:55:06.990769 4591 tablet_copy_client.cc:538] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261118 02:55:06.993625 4591 tablet_bootstrap.cc:492] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:06.999177 4591 tablet_bootstrap.cc:492] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:06.999537 4591 tablet_bootstrap.cc:492] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: Bootstrap complete.
I20261118 02:55:06.999789 4591 ts_tablet_manager.cc:1449] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20261118 02:55:07.001077 4591 raft_consensus.cc:360] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:07.001253 4591 raft_consensus.cc:748] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:07.001508 4591 consensus_queue.cc:261] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:07.002025 4591 ts_tablet_manager.cc:1480] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:07.003533 3775 tablet_copy_service.cc:342] P 99a39b1d9d814deeb3a8a644e9b392ef: Request end of tablet copy session 3f1973986ec1466eab5bec01dd81654f-ccf78258c9634540a178855a142f4d4b received from {username='slave'} at 127.2.229.67:42005
I20261118 02:55:07.003810 3775 tablet_copy_service.cc:434] P 99a39b1d9d814deeb3a8a644e9b392ef: ending tablet copy session 3f1973986ec1466eab5bec01dd81654f-ccf78258c9634540a178855a142f4d4b on tablet ccf78258c9634540a178855a142f4d4b with peer 3f1973986ec1466eab5bec01dd81654f
I20261118 02:55:07.062830 4594 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:07.063031 4594 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:07.064950 4594 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:07.065847 4459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:07.067314 4460 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
W20261118 02:55:07.067639 4459 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:07.067819 4459 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d; no voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:07.068499 4594 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20261118 02:55:07.068884 4580 raft_consensus.cc:499] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:07.069039 4580 raft_consensus.cc:521] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:07.069810 4580 leader_election.cc:290] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:07.070294 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ab61b43e639a4e75bc2e7009ea86a64f" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:07.070519 3901 raft_consensus.cc:2490] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:07.070547 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ab61b43e639a4e75bc2e7009ea86a64f" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:07.070770 3616 raft_consensus.cc:2490] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:07.070971 3690 leader_election.cc:304] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:07.071256 4580 raft_consensus.cc:2826] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:07.071375 4580 raft_consensus.cc:499] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:07.071467 4580 raft_consensus.cc:3096] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:07.073422 4580 raft_consensus.cc:521] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:07.073949 4580 leader_election.cc:290] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:07.074515 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ab61b43e639a4e75bc2e7009ea86a64f" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:07.074532 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "ab61b43e639a4e75bc2e7009ea86a64f" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:07.074702 3901 raft_consensus.cc:3096] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:07.074720 3616 raft_consensus.cc:3096] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:07.077730 3616 raft_consensus.cc:2490] T ab61b43e639a4e75bc2e7009ea86a64f P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:07.077744 3901 raft_consensus.cc:2490] T ab61b43e639a4e75bc2e7009ea86a64f P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:07.078269 3690 leader_election.cc:304] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:07.078702 4580 raft_consensus.cc:2826] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:07.078831 4580 raft_consensus.cc:705] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:07.079126 4580 consensus_queue.cc:238] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:07.082173 4580 consensus_queue.cc:1049] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:07.082392 4379 consensus_queue.cc:1049] T ab61b43e639a4e75bc2e7009ea86a64f P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:07.397850 4536 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:07.397953 4536 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: loading cluster ID for follower catalog manager: success
I20261118 02:55:07.401039 4536 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: acquiring CA information for follower catalog manager: success
I20261118 02:55:07.403229 4536 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:55:07.798398 4608 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:07.798589 4608 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:07.799456 4608 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:07.801621 4459 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:07.801791 4460 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:07.801931 4460 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d; no voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:07.802199 4608 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20261118 02:55:07.811005 3901 raft_consensus.cc:1227] T ccf78258c9634540a178855a142f4d4b P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20261118 02:55:08.838246 4620 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:08.838436 4620 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:08.839131 4620 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:08.841564 4460 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
W20261118 02:55:08.841743 4459 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:08.841873 4459 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d; no voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:08.842139 4620 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20261118 02:55:09.041154 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.125:38339
--webserver_interface=127.2.229.125
--webserver_port=39093
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 02:55:09.389125 4625 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:55:09.389431 4625 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:55:09.389555 4625 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:55:09.398095 4625 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:55:09.398192 4625 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:55:09.398248 4625 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:55:09.398312 4625 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:55:09.408986 4625 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.125:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.229.125
--webserver_port=39093
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:55:09.410651 4625 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:55:09.412458 4625 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:55:09.420225 4636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:09.420219 4635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:09.420872 4638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:55:09.421380 4625 server_base.cc:1061] running on GCE node
I20261118 02:55:09.422025 4625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:55:09.423336 4625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:55:09.424528 4625 hybrid_clock.cc:648] HybridClock initialized: now 1794970509424478 us; error 45 us; skew 500 ppm
I20261118 02:55:09.424825 4625 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:55:09.427099 4625 webserver.cc:577] Webserver started at http://127.2.229.125:39093/ using document root <none> and password file <none>
I20261118 02:55:09.427888 4625 fs_manager.cc:362] Metadata directory not provided
I20261118 02:55:09.428033 4625 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:55:09.428408 4643 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:09.428597 4643 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:09.429253 4643 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:09.431502 4459 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:09.431672 4460 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:09.431792 4460 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d; no voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:09.432049 4643 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20261118 02:55:09.436148 4625 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
I20261118 02:55:09.439002 4648 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:55:09.440493 4625 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20261118 02:55:09.440631 4625 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
uuid: "457eb38cba564ecd9f6f11245172260b"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:55:09.441294 4625 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:55:09.501787 4625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:55:09.502708 4625 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:55:09.503031 4625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:55:09.523041 4625 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.125:38339
I20261118 02:55:09.523105 4699 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.125:38339 every 8 connection(s)
I20261118 02:55:09.524816 4625 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
I20261118 02:55:09.531870 4700 sys_catalog.cc:263] Verifying existing consensus state
I20261118 02:55:09.531965 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 4625
I20261118 02:55:09.536155 4700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap starting.
I20261118 02:55:09.558239 4700 log.cc:826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:09.610821 3665 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:09.632992 4700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=93 ignored=0} inserts{seen=63 ignored=0} mutations{seen=79 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:09.633548 4700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap complete.
I20261118 02:55:09.638876 4700 raft_consensus.cc:360] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:09.639570 4700 raft_consensus.cc:748] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457eb38cba564ecd9f6f11245172260b, State: Initialized, Role: FOLLOWER
I20261118 02:55:09.640215 4700 consensus_queue.cc:261] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 4.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:09.640946 4706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:09.641184 4706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:09.641933 4700 sys_catalog.cc:565] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:55:09.661449 4720 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:09.661569 4720 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: loading cluster ID for follower catalog manager: success
I20261118 02:55:09.667950 4720 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: acquiring CA information for follower catalog manager: success
I20261118 02:55:09.670645 4720 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:55:09.685621 4625 master_runner.cc:187] Error getting master registration for 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111),
I20261118 02:55:09.691267 4625 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:09.836591 3949 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:09.838797 3802 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:09.839545 4665 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" instance_seqno: 1794970499527172) as {username='slave'} at 127.2.229.67:60229; Asking this server to re-register.
I20261118 02:55:09.840391 3949 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:09.841492 4665 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" instance_seqno: 1794970499028671) as {username='slave'} at 127.2.229.66:55747; Asking this server to re-register.
I20261118 02:55:09.842839 4665 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:09.843436 3802 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:09.845438 4664 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:10.078824 4706 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:10.079095 4706 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:10.081084 4706 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:10.081784 4484 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 5 candidate_status { last_received { term: 4 index: 93 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
W20261118 02:55:10.082010 4651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:10.082119 4484 raft_consensus.cc:2415] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 457eb38cba564ecd9f6f11245172260b in current term 5: Already voted for candidate 813a6937cea442ef81645fccc32207f4 in this term.
W20261118 02:55:10.083340 4651 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:10.083487 4651 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:10.083843 4706 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 4 FOLLOWER]: Advancing to term 5
I20261118 02:55:10.087531 4706 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261118 02:55:10.116843 3665 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:10.119119 4664 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" instance_seqno: 1794970498532688) as {username='slave'} at 127.2.229.65:45613; Asking this server to re-register.
I20261118 02:55:10.119891 3665 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:10.121568 4664 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:10.608711 4730 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:10.608884 4730 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:10.609583 4730 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:10.610154 4484 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 6 candidate_status { last_received { term: 4 index: 93 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:55:10.610455 4484 raft_consensus.cc:2432] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 457eb38cba564ecd9f6f11245172260b for term 6 because replica has last-logged OpId of term: 5 index: 97, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 93.
W20261118 02:55:10.612221 4651 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:10.612418 4651 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:10.613039 4730 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20261118 02:55:11.380443 4738 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:11.380635 4738 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:11.381322 4738 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:11.381956 4484 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 6 candidate_status { last_received { term: 4 index: 93 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:55:11.382246 4484 raft_consensus.cc:2432] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 457eb38cba564ecd9f6f11245172260b for term 6 because replica has last-logged OpId of term: 5 index: 97, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 93.
W20261118 02:55:11.383311 4651 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:11.383459 4651 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:11.383692 4738 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20261118 02:55:11.442483 4441 master_runner.cc:187] Error getting master registration for 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111),
I20261118 02:55:11.442680 4441 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:11.547122 4743 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:11.547319 4743 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:11.548496 4743 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:11.548717 4675 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 6 candidate_status { last_received { term: 5 index: 97 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b" is_pre_election: true
I20261118 02:55:11.549162 4675 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 5.
I20261118 02:55:11.549705 4459 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b; no voters:
W20261118 02:55:11.549947 4460 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:11.549986 4743 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20261118 02:55:11.550318 4743 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:11.550421 4743 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 5 FOLLOWER]: Advancing to term 6
I20261118 02:55:11.553462 4743 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:11.554422 4743 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 election: Requested vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:11.554540 4675 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 6 candidate_status { last_received { term: 5 index: 97 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b"
I20261118 02:55:11.554720 4675 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 5 FOLLOWER]: Advancing to term 6
W20261118 02:55:11.555732 4460 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:11.557613 4675 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 6.
I20261118 02:55:11.558022 4459 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b; no voters: 813a6937cea442ef81645fccc32207f4
I20261118 02:55:11.558318 4743 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 6 FOLLOWER]: Leader election won for term 6
I20261118 02:55:11.558694 4743 raft_consensus.cc:705] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 6 LEADER]: Becoming Leader. State: Replica: 26c6b20fa203404a88463ea27bdd3c7d, State: Running, Role: LEADER
I20261118 02:55:11.559206 4743 consensus_queue.cc:238] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 5.97, Last appended by leader: 97, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:11.561024 4745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 6 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:11.561278 4745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:11.566282 4748 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
W20261118 02:55:11.566767 4460 consensus_peers.cc:597] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d -> Peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Couldn't send request to peer 813a6937cea442ef81645fccc32207f4. Status: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:11.569782 4675 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Refusing update from remote peer 26c6b20fa203404a88463ea27bdd3c7d: Log matching property violated. Preceding OpId in replica: term: 4 index: 93. Preceding OpId from leader: term: 6 index: 98. (index mismatch)
I20261118 02:55:11.569769 4748 catalog_manager.cc:694] Loaded metadata for table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190]
I20261118 02:55:11.570315 4748 catalog_manager.cc:694] Loaded metadata for table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c]
I20261118 02:55:11.570602 4747 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 93, Time since last communication: 0.000s
I20261118 02:55:11.570757 4748 catalog_manager.cc:694] Loaded metadata for table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]
I20261118 02:55:11.571241 4748 catalog_manager.cc:694] Loaded metadata for table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]
I20261118 02:55:11.571622 4748 catalog_manager.cc:694] Loaded metadata for table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]
I20261118 02:55:11.572063 4748 catalog_manager.cc:694] Loaded metadata for table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]
I20261118 02:55:11.572690 4748 catalog_manager.cc:694] Loaded metadata for table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da]
I20261118 02:55:11.573125 4748 catalog_manager.cc:694] Loaded metadata for table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]
I20261118 02:55:11.573521 4748 catalog_manager.cc:694] Loaded metadata for table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d]
I20261118 02:55:11.573925 4748 catalog_manager.cc:694] Loaded metadata for table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]
I20261118 02:55:11.574307 4748 catalog_manager.cc:694] Loaded metadata for table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]
I20261118 02:55:11.574739 4748 catalog_manager.cc:694] Loaded metadata for table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b]
I20261118 02:55:11.575223 4748 catalog_manager.cc:694] Loaded metadata for table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]
I20261118 02:55:11.575758 4748 catalog_manager.cc:694] Loaded metadata for table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]
I20261118 02:55:11.576308 4748 catalog_manager.cc:694] Loaded metadata for table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]
I20261118 02:55:11.576771 4748 catalog_manager.cc:694] Loaded metadata for table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]
I20261118 02:55:11.577324 4748 catalog_manager.cc:694] Loaded metadata for table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]
I20261118 02:55:11.577795 4748 catalog_manager.cc:694] Loaded metadata for table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1]
I20261118 02:55:11.578333 4748 catalog_manager.cc:694] Loaded metadata for table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f]
I20261118 02:55:11.578850 4748 catalog_manager.cc:694] Loaded metadata for table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]
I20261118 02:55:11.578821 4738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 6 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:11.579099 4738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:11.579442 4751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:11.579715 4751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:11.580889 4749 mvcc.cc:204] Tried to move back new op lower bound from 7352199194737414144 to 7352199194150617088. Current Snapshot: MvccSnapshot[applied={T|T < 7352199194737414144 or (T in {7352199194737414144})}]
I20261118 02:55:11.581488 4748 tablet_loader.cc:96] loaded metadata for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7])
I20261118 02:55:11.581549 4743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:11.581795 4743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:11.581969 4748 tablet_loader.cc:96] loaded metadata for tablet 0e901a03b90e4c978eb98be87163cfe5 (table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190])
I20261118 02:55:11.582358 4748 tablet_loader.cc:96] loaded metadata for tablet 1904140da78b4567a9decb250c6f74a3 (table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f])
I20261118 02:55:11.582676 4748 tablet_loader.cc:96] loaded metadata for tablet 1d347c0ae53241708ec623562f4740d4 (table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b])
I20261118 02:55:11.582904 4751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:11.583074 4751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:11.583110 4748 tablet_loader.cc:96] loaded metadata for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db])
I20261118 02:55:11.583428 4748 tablet_loader.cc:96] loaded metadata for tablet 2c92955aa65142f1bd20f4b8f3321500 (table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d])
I20261118 02:55:11.584581 4748 tablet_loader.cc:96] loaded metadata for tablet 5f05a8e5b21945a5bd3df86a5467a615 (table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf])
I20261118 02:55:11.584870 4748 tablet_loader.cc:96] loaded metadata for tablet 65bd6fcaed5b47c3a2478d375ace9636 (table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73])
I20261118 02:55:11.585318 4748 tablet_loader.cc:96] loaded metadata for tablet 72f427c72a554ac1a0a2f2c3a78865da (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b])
I20261118 02:55:11.585609 4748 tablet_loader.cc:96] loaded metadata for tablet 78189cccd457466bb5d01231038fec24 (table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da])
I20261118 02:55:11.585911 4748 tablet_loader.cc:96] loaded metadata for tablet 8bed657d0da64d95851f8a86216b6c2c (table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c])
I20261118 02:55:11.586593 4748 tablet_loader.cc:96] loaded metadata for tablet a013d180f2cd430ab1ba30b39fd27591 (table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d])
I20261118 02:55:11.586977 4748 tablet_loader.cc:96] loaded metadata for tablet ab61b43e639a4e75bc2e7009ea86a64f (table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67])
I20261118 02:55:11.587219 4748 tablet_loader.cc:96] loaded metadata for tablet b129518ce23c45758ebcaa55410757d8 (table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1])
I20261118 02:55:11.587445 4748 tablet_loader.cc:96] loaded metadata for tablet b610080fc91745c7be260699946d5681 (table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139])
I20261118 02:55:11.587688 4748 tablet_loader.cc:96] loaded metadata for tablet c0713dcf79f54873a99d7556ad8f1454 (table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec])
I20261118 02:55:11.587950 4748 tablet_loader.cc:96] loaded metadata for tablet c92c29c30ab6411dabe6f02331d29b7a (table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d])
I20261118 02:55:11.588243 4748 tablet_loader.cc:96] loaded metadata for tablet ccf78258c9634540a178855a142f4d4b (table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e])
I20261118 02:55:11.588639 4748 tablet_loader.cc:96] loaded metadata for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6])
I20261118 02:55:11.589241 4748 tablet_loader.cc:96] loaded metadata for tablet ffe8d4c111d14054b76f9c7724072a94 (table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66])
I20261118 02:55:11.589478 4748 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 02:55:11.590610 4748 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:11.590677 4748 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:55:11.592012 4748 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:55:11.592973 4748 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Loaded TSK: 0
I20261118 02:55:11.593789 4748 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 02:55:11.594781 4748 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:55:11.595840 4748 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: 20
I20261118 02:55:11.667223 4472 master_service.cc:965] ReplaceTablet: received request to replace tablet b610080fc91745c7be260699946d5681 from {username='slave'} at 127.0.0.1:46014
I20261118 02:55:11.667469 4473 master_service.cc:965] ReplaceTablet: received request to replace tablet 5f05a8e5b21945a5bd3df86a5467a615 from {username='slave'} at 127.0.0.1:46014
I20261118 02:55:11.668659 4467 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46014:
table { table_name: "default.Table_a14cb6414555446286bf35bc5c6fb910" } new_table_name: "default.Table_324a2600fef947748ef03e27f7cb7e09" modify_external_catalogs: true
I20261118 02:55:11.668612 4468 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46014:
name: "default.Table_fd4fcac1f1d946fa898176f1d5e87e9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:11.669991 4474 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46014:
table { table_name: "default.Table_a7f3710d39fb447da94e16d4ab3933fd" } modify_external_catalogs: true
I20261118 02:55:11.670266 4474 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46014:
table { table_name: "default.Table_a7f3710d39fb447da94e16d4ab3933fd" } modify_external_catalogs: true
I20261118 02:55:11.671231 4469 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46014:
table { table_name: "default.Table_28c62f411da74545aa42dad7325bc064" } modify_external_catalogs: true
I20261118 02:55:11.671770 4469 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46014:
table { table_name: "default.Table_28c62f411da74545aa42dad7325bc064" } modify_external_catalogs: true
I20261118 02:55:11.670744 4471 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46014:
table { table_name: "default.Table_4f7796d5d5d54def843b971e66d15ea6" } new_table_name: "default.Table_79eb11797e3f4bc2bffe161096300f7a" modify_external_catalogs: true
I20261118 02:55:11.671438 4465 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46014:
name: "default.Table_dc982811b980416484416db1f74c685d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:11.674870 4466 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46014:
table { table_name: "default.Table_8ae376d4d4d64414b5c08dc3c2e58bc4" } new_table_name: "default.Table_5091d82298ca44bea5335d4989eb02d8" modify_external_catalogs: true
I20261118 02:55:11.673579 4470 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46014:
name: "default.Table_e39f59e2fdbb47fda4c10beba29ad8f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20261118 02:55:11.677047 4535 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 21, previous 21; dumping first 2 out of 15 received events: NotificationEvent(eventId=21, eventTime=1794970503, eventType=CREATE_TABLE, dbName=default, tableName=table_f0ae490a0df64e1097e144c7a370df61, message={"server":"","servicePrincipal":"","db":"default","table":"table_f0ae490a0df64e1097e144c7a370df61","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f0ae490a0df64e1097e144c7a370df61\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970503},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_f0ae490a0df64e1097e144c7a370df61\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f0ae490a0df64e1097e144c7a370df61\",\"kudu.table_id\":\"a59b9425c01049269883cf7767936c67\",\"transient_lastDdlTime\":\"1794970503\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1794970503, eventType=DROP_TABLE, dbName=default, tableName=table_c113c2b57e17479e92e241234a241af7, message={"server":"","servicePrincipal":"","db":"default","table":"table_c113c2b57e17479e92e241234a241af7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c113c2b57e17479e92e241234a241af7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970500},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_c113c2b57e17479e92e241234a241af7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c113c2b57e17479e92e241234a241af7\",\"kudu.table_id\":\"da0cb25ccfed426ca61f0dd9f93d30ec\",\"transient_lastDdlTime\":\"1794970500\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f436c3add5d google::LogMessage::Fail() at ??:0
@ 0x7f436c3b1b23 google::LogMessage::SendToLog() at ??:0
@ 0x7f436c3ad75c google::LogMessage::Flush() at ??:0
I20261118 02:55:11.686000 4473 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet 5f05a8e5b21945a5bd3df86a5467a615
@ 0x7f436c3ad9a1 google::LogMessage::~LogMessage() at ??:0
I20261118 02:55:11.688660 4473 catalog_manager.cc:6936] ReplaceTablet: tablet 5f05a8e5b21945a5bd3df86a5467a615 deleted and replaced by tablet 46f05790dbdc4b8185a7b0362f604cf8
I20261118 02:55:11.689280 4472 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet b610080fc91745c7be260699946d5681
I20261118 02:55:11.689844 4472 catalog_manager.cc:6936] ReplaceTablet: tablet b610080fc91745c7be260699946d5681 deleted and replaced by tablet 973831392ec24e0681c3d4fffd8bb199
@ 0x7f438182d9a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 02:55:11.710685 3595 tablet_service.cc:1547] Processing DeleteTablet for tablet b610080fc91745c7be260699946d5681 with delete_type TABLET_DATA_DELETED (replaced by tablet 973831392ec24e0681c3d4fffd8bb199) from {username='slave'} at 127.0.0.1:33686
I20261118 02:55:11.710644 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet 5f05a8e5b21945a5bd3df86a5467a615 with delete_type TABLET_DATA_DELETED (replaced by tablet 46f05790dbdc4b8185a7b0362f604cf8) from {username='slave'} at 127.0.0.1:33686
I20261118 02:55:11.714581 4473 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46014:
table { table_name: "default.Table_0385cef6593c419eb426657c8cbef2bf" } new_table_name: "default.Table_496441d99faa4e2b8c7cd98e4a5ce745" modify_external_catalogs: true
@ 0x7f438182c251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20261118 02:55:11.719249 4472 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46014:
name: "default.Table_51d36ff687ca47b5995e5693d7b8931b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f4381833ecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20261118 02:55:11.720479 4766 tablet_replica.cc:333] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:11.720708 4766 raft_consensus.cc:2261] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:11.721908 4766 raft_consensus.cc:2290] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:11.722530 4766 ts_tablet_manager.cc:1962] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f4381833d49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
I20261118 02:55:11.726418 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet 5f05a8e5b21945a5bd3df86a5467a615 with delete_type TABLET_DATA_DELETED (replaced by tablet 46f05790dbdc4b8185a7b0362f604cf8) from {username='slave'} at 127.0.0.1:56640
I20261118 02:55:11.726590 3879 tablet_service.cc:1547] Processing DeleteTablet for tablet b610080fc91745c7be260699946d5681 with delete_type TABLET_DATA_DELETED (replaced by tablet 973831392ec24e0681c3d4fffd8bb199) from {username='slave'} at 127.0.0.1:56640
I20261118 02:55:11.727041 4767 tablet_replica.cc:333] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:11.727229 4767 raft_consensus.cc:2261] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:11.727582 4767 raft_consensus.cc:2290] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:11.728190 4767 ts_tablet_manager.cc:1962] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:11.729379 3881 tablet_service.cc:1500] Processing CreateTablet for tablet 973831392ec24e0681c3d4fffd8bb199 (DEFAULT_TABLE table=default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:11.729784 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973831392ec24e0681c3d4fffd8bb199. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:11.730211 4766 ts_tablet_manager.cc:1975] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:11.730320 4766 log.cc:1199] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/b610080fc91745c7be260699946d5681
I20261118 02:55:11.730760 4766 ts_tablet_manager.cc:1996] T b610080fc91745c7be260699946d5681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:11.731588 3595 tablet_service.cc:1500] Processing CreateTablet for tablet 973831392ec24e0681c3d4fffd8bb199 (DEFAULT_TABLE table=default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:11.731954 3595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973831392ec24e0681c3d4fffd8bb199. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:11.732034 3879 tablet_service.cc:1500] Processing CreateTablet for tablet 46f05790dbdc4b8185a7b0362f604cf8 (DEFAULT_TABLE table=default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:11.732493 3879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f05790dbdc4b8185a7b0362f604cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:11.733669 3597 tablet_service.cc:1500] Processing CreateTablet for tablet 46f05790dbdc4b8185a7b0362f604cf8 (DEFAULT_TABLE table=default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:11.734030 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f05790dbdc4b8185a7b0362f604cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:11.734999 4767 ts_tablet_manager.cc:1975] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:11.735173 4767 log.cc:1199] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/5f05a8e5b21945a5bd3df86a5467a615
I20261118 02:55:11.735579 4767 ts_tablet_manager.cc:1996] T 5f05a8e5b21945a5bd3df86a5467a615 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:11.736608 4767 tablet_replica.cc:333] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:11.736804 4767 raft_consensus.cc:2261] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:11.736778 4460 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet 5f05a8e5b21945a5bd3df86a5467a615 (table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]) successfully deleted
I20261118 02:55:11.736963 4767 raft_consensus.cc:2290] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:11.737495 4767 ts_tablet_manager.cc:1962] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:11.738997 4461 catalog_manager.cc:5065] TS 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): tablet b610080fc91745c7be260699946d5681 (table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]) successfully deleted
I20261118 02:55:11.739292 4766 tablet_replica.cc:333] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:11.739506 4766 raft_consensus.cc:2261] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:11.739673 4766 raft_consensus.cc:2290] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:11.740213 4766 ts_tablet_manager.cc:1962] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:11.741859 3734 tablet_service.cc:1547] Processing DeleteTablet for tablet 5f05a8e5b21945a5bd3df86a5467a615 with delete_type TABLET_DATA_DELETED (replaced by tablet 46f05790dbdc4b8185a7b0362f604cf8) from {username='slave'} at 127.0.0.1:48156
I20261118 02:55:11.742293 3735 tablet_service.cc:1547] Processing DeleteTablet for tablet b610080fc91745c7be260699946d5681 with delete_type TABLET_DATA_DELETED (replaced by tablet 973831392ec24e0681c3d4fffd8bb199) from {username='slave'} at 127.0.0.1:48156
I20261118 02:55:11.742537 3732 tablet_service.cc:1500] Processing CreateTablet for tablet 46f05790dbdc4b8185a7b0362f604cf8 (DEFAULT_TABLE table=default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:11.742699 3733 tablet_service.cc:1500] Processing CreateTablet for tablet 973831392ec24e0681c3d4fffd8bb199 (DEFAULT_TABLE table=default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:11.742998 3732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f05790dbdc4b8185a7b0362f604cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:11.743232 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973831392ec24e0681c3d4fffd8bb199. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:11.743693 4767 ts_tablet_manager.cc:1975] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:11.743825 4767 log.cc:1199] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/b610080fc91745c7be260699946d5681
I20261118 02:55:11.744275 4767 ts_tablet_manager.cc:1996] T b610080fc91745c7be260699946d5681 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:11.745342 4460 catalog_manager.cc:5065] TS 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): tablet b610080fc91745c7be260699946d5681 (table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]) successfully deleted
@ 0x7f436d7f6dc9 kudu::Thread::SuperviseThread() at ??:0
I20261118 02:55:11.753643 4772 tablet_bootstrap.cc:492] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
@ 0x7f4370ed16db start_thread at ??:0
@ 0x7f436994161f clone at ??:0
I20261118 02:55:11.756464 4772 tablet_bootstrap.cc:654] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:11.756669 4770 tablet_replica.cc:333] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:11.757047 4770 raft_consensus.cc:2261] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:11.757342 4770 raft_consensus.cc:2290] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:11.758003 4770 ts_tablet_manager.cc:1962] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:11.758157 4768 tablet_bootstrap.cc:492] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:11.760226 4768 tablet_bootstrap.cc:654] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:11.760560 4769 tablet_bootstrap.cc:492] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:11.762404 4769 tablet_bootstrap.cc:654] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:11.762938 4768 tablet_bootstrap.cc:492] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:11.763072 4768 ts_tablet_manager.cc:1449] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261118 02:55:11.764050 4772 tablet_bootstrap.cc:492] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:11.764220 4772 ts_tablet_manager.cc:1449] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20261118 02:55:11.764508 4770 ts_tablet_manager.cc:1975] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:11.764614 4770 log.cc:1199] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/5f05a8e5b21945a5bd3df86a5467a615
I20261118 02:55:11.765028 4770 ts_tablet_manager.cc:1996] T 5f05a8e5b21945a5bd3df86a5467a615 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:11.765053 4772 raft_consensus.cc:360] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:11.765252 4772 raft_consensus.cc:386] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:11.765337 4772 raft_consensus.cc:748] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:11.765617 4772 consensus_queue.cc:261] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:11.765971 4770 tablet_replica.cc:333] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:11.767235 4772 ts_tablet_manager.cc:1480] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20261118 02:55:11.767527 4772 tablet_bootstrap.cc:492] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:11.769248 4772 tablet_bootstrap.cc:654] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:11.769541 4766 ts_tablet_manager.cc:1975] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:11.769680 4766 log.cc:1199] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/5f05a8e5b21945a5bd3df86a5467a615
I20261118 02:55:11.770109 4766 ts_tablet_manager.cc:1996] T 5f05a8e5b21945a5bd3df86a5467a615 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:11.771042 4768 raft_consensus.cc:360] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:11.771230 4768 raft_consensus.cc:386] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:11.771314 4768 raft_consensus.cc:748] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:11.771592 4768 consensus_queue.cc:261] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:11.772011 3663 heartbeater.cc:500] Master 127.2.229.126:38025 was elected leader, sending a full tablet report...
I20261118 02:55:11.772783 4770 raft_consensus.cc:2261] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:11.774117 4768 ts_tablet_manager.cc:1480] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20261118 02:55:11.774374 4768 tablet_bootstrap.cc:492] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:11.775724 4769 tablet_bootstrap.cc:492] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:11.775887 4769 ts_tablet_manager.cc:1449] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20261118 02:55:11.777119 4769 raft_consensus.cc:360] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:11.777314 4769 raft_consensus.cc:386] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:11.777390 4769 raft_consensus.cc:748] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:11.777617 4769 consensus_queue.cc:261] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:11.778069 4768 tablet_bootstrap.cc:654] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:11.779294 4770 raft_consensus.cc:2290] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:11.781677 4770 ts_tablet_manager.cc:1962] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:11.782068 4769 ts_tablet_manager.cc:1480] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20261118 02:55:11.784134 4769 tablet_bootstrap.cc:492] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:11.787536 4769 tablet_bootstrap.cc:654] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:11.790158 4770 ts_tablet_manager.cc:1975] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:11.790294 4770 log.cc:1199] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/b610080fc91745c7be260699946d5681
I20261118 02:55:11.790688 4772 tablet_bootstrap.cc:492] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:11.790719 4770 ts_tablet_manager.cc:1996] T b610080fc91745c7be260699946d5681 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:11.790863 4772 ts_tablet_manager.cc:1449] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.005s
I20261118 02:55:11.791970 4772 raft_consensus.cc:360] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:11.792196 4772 raft_consensus.cc:386] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:11.792260 4772 raft_consensus.cc:748] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:11.792502 4772 consensus_queue.cc:261] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:11.793334 4772 ts_tablet_manager.cc:1480] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:11.795405 4768 tablet_bootstrap.cc:492] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:11.795593 4768 ts_tablet_manager.cc:1449] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.008s
I20261118 02:55:11.795606 4769 tablet_bootstrap.cc:492] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:11.795755 4769 ts_tablet_manager.cc:1449] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20261118 02:55:11.796408 4768 raft_consensus.cc:360] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:11.796617 4768 raft_consensus.cc:386] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:11.796782 4768 raft_consensus.cc:748] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:11.797115 4768 consensus_queue.cc:261] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:11.797746 2965 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task8fD3U4/build/asan/bin/kudu pid 4625
I20261118 02:55:11.796482 4769 raft_consensus.cc:360] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:11.798854 4769 raft_consensus.cc:386] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:11.798943 4769 raft_consensus.cc:748] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:11.799081 4768 ts_tablet_manager.cc:1480] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:11.799238 4769 consensus_queue.cc:261] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:11.803267 4769 ts_tablet_manager.cc:1480] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20261118 02:55:11.808547 4695 generic_service.cc:196] Checking for leaks (request via RPC)
I20261118 02:55:11.921391 2965 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task8fD3U4/build/asan/bin/kudu with pid 4625
W20261118 02:55:11.958244 3837 connection.cc:573] server connection from 127.0.0.1:56640 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:11.958298 3689 connection.cc:573] server connection from 127.0.0.1:48156 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:11.958316 3551 connection.cc:573] server connection from 127.0.0.1:33686 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:11.958477 3956 connection.cc:573] client connection to 127.2.229.126:38025 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:11.958549 3837 connection.cc:573] client connection to 127.2.229.126:38025 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:11.958592 3690 connection.cc:573] client connection to 127.2.229.126:38025 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:11.958904 3553 connection.cc:573] client connection to 127.2.229.126:38025 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:11.959170 3956 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.229.126:38025)
W20261118 02:55:11.959489 3956 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.229.126:38025)
W20261118 02:55:11.959905 3956 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.229.126:38025)
W20261118 02:55:11.960073 3956 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.229.126:38025)
W20261118 02:55:11.961609 4756 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:11.959989 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.2.229.126:38025 (local address 127.2.229.65:58997)
1118 02:55:11.960252 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1118 02:55:11.960270 (+ 18us) client_negotiation.cc:175] Beginning negotiation
1118 02:55:11.960398 (+ 128us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 02:55:11.961565 (+ 1167us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":164}
W20261118 02:55:11.962599 4736 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:11.961274 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.2.229.126:38025 (local address 127.2.229.66:60635)
1118 02:55:11.961579 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1118 02:55:11.961594 (+ 15us) client_negotiation.cc:175] Beginning negotiation
1118 02:55:11.961695 (+ 101us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 02:55:11.962556 (+ 861us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":185}
W20261118 02:55:11.962687 4737 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:11.960595 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.2.229.126:38025 (local address 127.2.229.67:45299)
1118 02:55:11.960835 (+ 240us) negotiation.cc:107] Waiting for socket to connect
1118 02:55:11.960852 (+ 17us) client_negotiation.cc:175] Beginning negotiation
1118 02:55:11.960967 (+ 115us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 02:55:11.962642 (+ 1675us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":142}
I20261118 02:55:12.013914 4581 raft_consensus.cc:499] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:12.014122 4581 raft_consensus.cc:521] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:12.014883 4581 leader_election.cc:290] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:12.015492 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "46f05790dbdc4b8185a7b0362f604cf8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:12.015506 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "46f05790dbdc4b8185a7b0362f604cf8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:12.015697 3901 raft_consensus.cc:2490] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:12.015723 3751 raft_consensus.cc:2490] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:12.016202 3553 leader_election.cc:304] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:12.016482 4581 raft_consensus.cc:2826] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:12.016603 4581 raft_consensus.cc:499] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:12.016687 4581 raft_consensus.cc:3096] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:12.018610 4581 raft_consensus.cc:521] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:12.019210 4581 leader_election.cc:290] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:12.019716 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "46f05790dbdc4b8185a7b0362f604cf8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:12.019703 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "46f05790dbdc4b8185a7b0362f604cf8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:12.019913 3901 raft_consensus.cc:3096] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:12.019930 3751 raft_consensus.cc:3096] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:12.022164 3751 raft_consensus.cc:2490] T 46f05790dbdc4b8185a7b0362f604cf8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:12.022176 3901 raft_consensus.cc:2490] T 46f05790dbdc4b8185a7b0362f604cf8 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:12.022570 3552 leader_election.cc:304] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:12.022851 4581 raft_consensus.cc:2826] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:12.023233 4581 raft_consensus.cc:705] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:12.023773 4581 consensus_queue.cc:238] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:12.031153 4794 consensus_queue.cc:1049] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:12.032168 4794 consensus_queue.cc:1049] T 46f05790dbdc4b8185a7b0362f604cf8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:12.229302 4614 raft_consensus.cc:499] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:12.229525 4614 raft_consensus.cc:521] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:12.230232 4614 leader_election.cc:290] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:12.230854 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "973831392ec24e0681c3d4fffd8bb199" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:12.230854 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "973831392ec24e0681c3d4fffd8bb199" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:12.231078 3616 raft_consensus.cc:2490] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:12.231088 3751 raft_consensus.cc:2490] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:12.231495 3836 leader_election.cc:304] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:12.231762 4614 raft_consensus.cc:2826] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:12.231863 4614 raft_consensus.cc:499] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:12.231945 4614 raft_consensus.cc:3096] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:12.233860 4614 raft_consensus.cc:521] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:12.234371 4614 leader_election.cc:290] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:12.234848 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "973831392ec24e0681c3d4fffd8bb199" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:12.234838 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "973831392ec24e0681c3d4fffd8bb199" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:12.235011 3616 raft_consensus.cc:3096] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:12.235010 3751 raft_consensus.cc:3096] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:12.236936 3616 raft_consensus.cc:2490] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:12.237143 3751 raft_consensus.cc:2490] T 973831392ec24e0681c3d4fffd8bb199 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:12.237317 3838 leader_election.cc:304] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:12.237659 4614 raft_consensus.cc:2826] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:12.238065 4614 raft_consensus.cc:705] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:12.239562 4614 consensus_queue.cc:238] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:12.244458 4805 consensus_queue.cc:1049] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:12.247144 4805 consensus_queue.cc:1049] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:12.541921 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.125:38339
--webserver_interface=127.2.229.125
--webserver_port=39093
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 02:55:12.885465 4810 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:55:12.885756 4810 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:55:12.885866 4810 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:55:12.894454 4810 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:55:12.894559 4810 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:55:12.894636 4810 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:55:12.894699 4810 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:55:12.904956 4810 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.125:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.229.125
--webserver_port=39093
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:55:12.906653 4810 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:55:12.908629 4810 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:55:12.916683 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:12.916708 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:12.917578 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:55:12.918057 4810 server_base.cc:1061] running on GCE node
I20261118 02:55:12.919044 4810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:55:12.920642 4810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:55:12.921864 4810 hybrid_clock.cc:648] HybridClock initialized: now 1794970512921809 us; error 46 us; skew 500 ppm
I20261118 02:55:12.922224 4810 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:55:12.924669 4810 webserver.cc:577] Webserver started at http://127.2.229.125:39093/ using document root <none> and password file <none>
I20261118 02:55:12.925457 4810 fs_manager.cc:362] Metadata directory not provided
I20261118 02:55:12.925565 4810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:55:12.931278 4810 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20261118 02:55:12.934192 4829 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:55:12.935770 4810 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:12.935956 4810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
uuid: "457eb38cba564ecd9f6f11245172260b"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:55:12.936721 4810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:55:12.957002 4810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:55:12.957846 4810 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:55:12.958112 4810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:55:12.976824 4810 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.125:38339
I20261118 02:55:12.976848 4880 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.125:38339 every 8 connection(s)
I20261118 02:55:12.978459 4810 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
I20261118 02:55:12.981709 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 4810
I20261118 02:55:12.986006 4881 sys_catalog.cc:263] Verifying existing consensus state
I20261118 02:55:12.989521 4881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap starting.
I20261118 02:55:13.007787 4881 log.cc:826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:13.090138 4881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap replayed 1/1 log segments. Stats: ops{read=101 overwritten=0 applied=101 ignored=0} inserts{seen=67 ignored=0} mutations{seen=87 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:13.090667 4881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap complete.
I20261118 02:55:13.096048 4881 raft_consensus.cc:360] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:13.096746 4881 raft_consensus.cc:748] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457eb38cba564ecd9f6f11245172260b, State: Initialized, Role: FOLLOWER
I20261118 02:55:13.097409 4881 consensus_queue.cc:261] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 101, Last appended: 6.101, Last appended by leader: 101, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:13.098317 4885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:13.098572 4885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:13.099340 4881 sys_catalog.cc:565] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:55:13.112406 4899 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:13.112562 4899 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: loading cluster ID for follower catalog manager: success
I20261118 02:55:13.107885 4810 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:55:13.113579 4810 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:13.117264 4899 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: acquiring CA information for follower catalog manager: success
I20261118 02:55:13.119794 4899 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:55:13.273941 3802 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:13.279958 4846 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" instance_seqno: 1794970499028671) as {username='slave'} at 127.2.229.66:45637; Asking this server to re-register.
I20261118 02:55:13.281105 3802 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:13.283844 4845 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:13.287583 3665 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:13.288973 3949 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:13.290493 4845 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" instance_seqno: 1794970498532688) as {username='slave'} at 127.2.229.65:57643; Asking this server to re-register.
I20261118 02:55:13.290880 4846 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" instance_seqno: 1794970499527172) as {username='slave'} at 127.2.229.67:34069; Asking this server to re-register.
I20261118 02:55:13.291088 3665 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:13.291548 3949 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:13.292868 4846 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:13.293329 4845 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:13.661460 4910 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:13.661701 4910 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:13.663705 4910 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:13.664698 4832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:13.666074 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:13.666324 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:13.666428 4832 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:13.666774 4910 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20261118 02:55:14.398300 4919 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:14.398475 4919 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:14.399085 4919 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:14.401381 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:14.401618 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:14.401734 4832 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:14.402002 4919 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20261118 02:55:15.200219 4927 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:15.200452 4927 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:15.201141 4927 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:15.203652 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:15.203918 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:15.204056 4832 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:15.204376 4927 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20261118 02:55:16.395913 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.124:36695
--webserver_interface=127.2.229.124
--webserver_port=39557
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 02:55:16.440620 4948 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:16.440841 4948 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:16.441515 4948 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:16.443845 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:16.444074 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:16.444254 4832 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:16.444526 4948 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20261118 02:55:16.739959 4947 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:55:16.740352 4947 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:55:16.740473 4947 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:55:16.749022 4947 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:55:16.749122 4947 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:55:16.749184 4947 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:55:16.749251 4947 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:55:16.759747 4947 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.124:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.229.124
--webserver_port=39557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:55:16.761472 4947 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:55:16.763316 4947 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:55:16.772500 4956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:16.772330 4955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:16.772840 4958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:55:16.774017 4947 server_base.cc:1061] running on GCE node
I20261118 02:55:16.774778 4947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:55:16.776278 4947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:55:16.777500 4947 hybrid_clock.cc:648] HybridClock initialized: now 1794970516777444 us; error 47 us; skew 500 ppm
I20261118 02:55:16.777848 4947 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:55:16.779922 4947 webserver.cc:577] Webserver started at http://127.2.229.124:39557/ using document root <none> and password file <none>
I20261118 02:55:16.780642 4947 fs_manager.cc:362] Metadata directory not provided
I20261118 02:55:16.780753 4947 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:55:16.786494 4947 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20261118 02:55:16.789310 4965 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:55:16.791010 4947 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261118 02:55:16.791225 4947 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
uuid: "813a6937cea442ef81645fccc32207f4"
format_stamp: "Formatted at 2026-11-18 02:54:57 on dist-test-slave-49p1"
I20261118 02:55:16.792274 4947 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:55:16.812065 4947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:55:16.812987 4947 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:55:16.813256 4947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:55:16.831676 4947 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.124:36695
I20261118 02:55:16.831676 5018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.124:36695 every 8 connection(s)
I20261118 02:55:16.833537 4947 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
I20261118 02:55:16.840436 5019 sys_catalog.cc:263] Verifying existing consensus state
I20261118 02:55:16.843278 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 4947
I20261118 02:55:16.843986 5019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap starting.
I20261118 02:55:16.862686 5019 log.cc:826] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:16.933223 5019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=97 ignored=0} inserts{seen=65 ignored=0} mutations{seen=83 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:16.933740 5019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap complete.
I20261118 02:55:16.938995 5019 raft_consensus.cc:360] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:16.939662 5019 raft_consensus.cc:748] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813a6937cea442ef81645fccc32207f4, State: Initialized, Role: FOLLOWER
I20261118 02:55:16.940313 5019 consensus_queue.cc:261] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 5.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:16.941090 5023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:16.941354 5023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:16.942106 5019 sys_catalog.cc:565] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:55:16.955051 5037 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:16.955246 5037 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: loading cluster ID for follower catalog manager: success
I20261118 02:55:16.951622 4947 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:55:16.957418 4947 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:16.959854 5037 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: acquiring CA information for follower catalog manager: success
I20261118 02:55:16.963867 5037 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:55:17.268033 5042 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:17.268275 5042 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:17.268959 5042 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:17.272802 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:17.277618 4994 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 7 candidate_status { last_received { term: 6 index: 101 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4" is_pre_election: true
I20261118 02:55:17.278187 4994 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 457eb38cba564ecd9f6f11245172260b in term 5.
I20261118 02:55:17.278980 4832 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d
I20261118 02:55:17.279316 5042 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20261118 02:55:17.279435 5042 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:17.279528 5042 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 6 FOLLOWER]: Advancing to term 7
I20261118 02:55:17.283109 5042 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:17.283646 5042 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 election: Requested vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:17.284798 4994 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 7 candidate_status { last_received { term: 6 index: 101 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4"
I20261118 02:55:17.284991 4994 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 5 FOLLOWER]: Advancing to term 7
W20261118 02:55:17.285351 4832 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:17.289671 4994 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 457eb38cba564ecd9f6f11245172260b in term 7.
I20261118 02:55:17.290100 4832 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d
I20261118 02:55:17.290369 5042 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 7 FOLLOWER]: Leader election won for term 7
I20261118 02:55:17.290776 5042 raft_consensus.cc:705] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 7 LEADER]: Becoming Leader. State: Replica: 457eb38cba564ecd9f6f11245172260b, State: Running, Role: LEADER
I20261118 02:55:17.291271 5042 consensus_queue.cc:238] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 6.101, Last appended by leader: 101, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:17.293905 3664 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:55:17.298641 4984 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" instance_seqno: 1794970498532688) as {username='slave'} at 127.2.229.65:48463; Asking this server to re-register.
I20261118 02:55:17.302032 4994 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Refusing update from remote peer 457eb38cba564ecd9f6f11245172260b: Log matching property violated. Preceding OpId in replica: term: 5 index: 97. Preceding OpId from leader: term: 7 index: 102. (index mismatch)
W20261118 02:55:17.304693 4832 consensus_peers.cc:597] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b -> Peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Couldn't send request to peer 26c6b20fa203404a88463ea27bdd3c7d. Status: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:17.304706 5042 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Connected to new peer: Peer: permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 97, Time since last communication: 0.000s
I20261118 02:55:17.318737 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 7 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:17.320247 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:17.325256 3664 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:17.328212 5056 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 02:55:17.335031 3948 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:55:17.335955 5056 catalog_manager.cc:694] Loaded metadata for table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190]
I20261118 02:55:17.336545 5056 catalog_manager.cc:694] Loaded metadata for table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c]
I20261118 02:55:17.342368 5056 catalog_manager.cc:694] Loaded metadata for table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]
I20261118 02:55:17.343005 5056 catalog_manager.cc:694] Loaded metadata for table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]
I20261118 02:55:17.344633 5056 catalog_manager.cc:694] Loaded metadata for table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]
I20261118 02:55:17.345050 5056 catalog_manager.cc:694] Loaded metadata for table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]
I20261118 02:55:17.345623 5056 catalog_manager.cc:694] Loaded metadata for table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da]
I20261118 02:55:17.346002 3801 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:55:17.346076 5056 catalog_manager.cc:694] Loaded metadata for table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]
I20261118 02:55:17.346498 5056 catalog_manager.cc:694] Loaded metadata for table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d]
I20261118 02:55:17.346930 5056 catalog_manager.cc:694] Loaded metadata for table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]
I20261118 02:55:17.347308 5056 catalog_manager.cc:694] Loaded metadata for table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]
I20261118 02:55:17.347702 5056 catalog_manager.cc:694] Loaded metadata for table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b]
I20261118 02:55:17.348191 5056 catalog_manager.cc:694] Loaded metadata for table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]
I20261118 02:55:17.348608 5056 catalog_manager.cc:694] Loaded metadata for table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]
I20261118 02:55:17.349020 5056 catalog_manager.cc:694] Loaded metadata for table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]
I20261118 02:55:17.349402 5056 catalog_manager.cc:694] Loaded metadata for table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]
I20261118 02:55:17.349769 5056 catalog_manager.cc:694] Loaded metadata for table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]
I20261118 02:55:17.350140 5056 catalog_manager.cc:694] Loaded metadata for table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1]
I20261118 02:55:17.350515 5056 catalog_manager.cc:694] Loaded metadata for table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f]
I20261118 02:55:17.350921 5056 catalog_manager.cc:694] Loaded metadata for table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]
I20261118 02:55:17.353554 5056 tablet_loader.cc:96] loaded metadata for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7])
I20261118 02:55:17.353971 5056 tablet_loader.cc:96] loaded metadata for tablet 0e901a03b90e4c978eb98be87163cfe5 (table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190])
I20261118 02:55:17.356763 5054 mvcc.cc:204] Tried to move back new op lower bound from 7352199215829995520 to 7352199215350587392. Current Snapshot: MvccSnapshot[applied={T|T < 7352199215829995520 or (T in {7352199215829995520})}]
I20261118 02:55:17.357405 5056 tablet_loader.cc:96] loaded metadata for tablet 1904140da78b4567a9decb250c6f74a3 (table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f])
I20261118 02:55:17.357864 5056 tablet_loader.cc:96] loaded metadata for tablet 1d347c0ae53241708ec623562f4740d4 (table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b])
I20261118 02:55:17.358242 5056 tablet_loader.cc:96] loaded metadata for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db])
I20261118 02:55:17.358572 5056 tablet_loader.cc:96] loaded metadata for tablet 2c92955aa65142f1bd20f4b8f3321500 (table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d])
I20261118 02:55:17.359119 5056 tablet_loader.cc:96] loaded metadata for tablet 46f05790dbdc4b8185a7b0362f604cf8 (table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf])
I20261118 02:55:17.360445 5056 tablet_loader.cc:96] loaded metadata for tablet 65bd6fcaed5b47c3a2478d375ace9636 (table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73])
I20261118 02:55:17.360956 5056 tablet_loader.cc:96] loaded metadata for tablet 72f427c72a554ac1a0a2f2c3a78865da (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b])
I20261118 02:55:17.361289 5056 tablet_loader.cc:96] loaded metadata for tablet 78189cccd457466bb5d01231038fec24 (table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da])
I20261118 02:55:17.361581 5056 tablet_loader.cc:96] loaded metadata for tablet 8bed657d0da64d95851f8a86216b6c2c (table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c])
I20261118 02:55:17.362082 5056 tablet_loader.cc:96] loaded metadata for tablet 973831392ec24e0681c3d4fffd8bb199 (table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139])
I20261118 02:55:17.362761 5056 tablet_loader.cc:96] loaded metadata for tablet a013d180f2cd430ab1ba30b39fd27591 (table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d])
I20261118 02:55:17.363284 5056 tablet_loader.cc:96] loaded metadata for tablet ab61b43e639a4e75bc2e7009ea86a64f (table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67])
I20261118 02:55:17.363675 5056 tablet_loader.cc:96] loaded metadata for tablet b129518ce23c45758ebcaa55410757d8 (table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1])
I20261118 02:55:17.364313 5056 tablet_loader.cc:96] loaded metadata for tablet c0713dcf79f54873a99d7556ad8f1454 (table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec])
I20261118 02:55:17.364655 5056 tablet_loader.cc:96] loaded metadata for tablet c92c29c30ab6411dabe6f02331d29b7a (table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d])
I20261118 02:55:17.365108 5056 tablet_loader.cc:96] loaded metadata for tablet ccf78258c9634540a178855a142f4d4b (table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e])
I20261118 02:55:17.365415 5023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 7 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:17.365669 5023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:17.367105 4983 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:17.367910 5023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:17.368116 5023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:17.368207 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:17.368510 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:17.369226 4981 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" instance_seqno: 1794970499028671) as {username='slave'} at 127.2.229.66:34351; Asking this server to re-register.
I20261118 02:55:17.370055 4982 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" instance_seqno: 1794970499527172) as {username='slave'} at 127.2.229.67:36637; Asking this server to re-register.
I20261118 02:55:17.370177 3801 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:17.370800 3948 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:17.371964 5056 tablet_loader.cc:96] loaded metadata for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6])
I20261118 02:55:17.372258 4982 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:17.372856 5056 tablet_loader.cc:96] loaded metadata for tablet ffe8d4c111d14054b76f9c7724072a94 (table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66])
I20261118 02:55:17.373199 5056 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 02:55:17.373705 4982 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:17.374462 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:17.374694 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:17.374845 5056 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:17.374914 5056 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:55:17.376365 5056 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:55:17.377336 5056 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Loaded TSK: 0
I20261118 02:55:17.378203 5056 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 02:55:17.379172 5056 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:55:17.380203 5056 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: 20
I20261118 02:55:17.478365 4845 master_service.cc:965] ReplaceTablet: received request to replace tablet c92c29c30ab6411dabe6f02331d29b7a from {username='slave'} at 127.0.0.1:42302
I20261118 02:55:17.478494 4846 master_service.cc:965] ReplaceTablet: received request to replace tablet 1d347c0ae53241708ec623562f4740d4 from {username='slave'} at 127.0.0.1:42302
I20261118 02:55:17.479507 4842 master_service.cc:965] ReplaceTablet: received request to replace tablet 1d347c0ae53241708ec623562f4740d4 from {username='slave'} at 127.0.0.1:42302
I20261118 02:55:17.479658 4840 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_87d592a09c3348028719a603956fbce6" } modify_external_catalogs: true
I20261118 02:55:17.480013 4840 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_87d592a09c3348028719a603956fbce6" } modify_external_catalogs: true
I20261118 02:55:17.479971 4839 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42302:
name: "default.Table_bdbed231860946c2bf9b514f724bc3e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:17.480463 4838 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42302:
name: "default.Table_40868189cecf45d0b705461c4328ee17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:17.481050 4837 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42302:
name: "default.Table_52fca7a2e65f4e679bbef9de3df569f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:17.482582 4841 master_service.cc:965] ReplaceTablet: received request to replace tablet 55f81e7b62e54ec18c7c13a200edb68f from {username='slave'} at 127.0.0.1:42302
I20261118 02:55:17.482751 4843 master_service.cc:965] ReplaceTablet: received request to replace tablet a013d180f2cd430ab1ba30b39fd27591 from {username='slave'} at 127.0.0.1:42302
I20261118 02:55:17.482508 4844 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_2e3917ff98f2474984580e61caa9c17b" } new_table_name: "default.Table_3c225da1f7fa435083909e053ec9751a" modify_external_catalogs: true
I20261118 02:55:17.483662 4841 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_571d1360811c42dc815efb439b2b52e6" } modify_external_catalogs: true
I20261118 02:55:17.483940 4841 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_571d1360811c42dc815efb439b2b52e6" } modify_external_catalogs: true
F20261118 02:55:17.488473 4898 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 21, previous 21; dumping first 2 out of 15 received events: NotificationEvent(eventId=21, eventTime=1794970503, eventType=CREATE_TABLE, dbName=default, tableName=table_f0ae490a0df64e1097e144c7a370df61, message={"server":"","servicePrincipal":"","db":"default","table":"table_f0ae490a0df64e1097e144c7a370df61","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f0ae490a0df64e1097e144c7a370df61\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970503},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_f0ae490a0df64e1097e144c7a370df61\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f0ae490a0df64e1097e144c7a370df61\",\"kudu.table_id\":\"a59b9425c01049269883cf7767936c67\",\"transient_lastDdlTime\":\"1794970503\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1794970503, eventType=DROP_TABLE, dbName=default, tableName=table_c113c2b57e17479e92e241234a241af7, message={"server":"","servicePrincipal":"","db":"default","table":"table_c113c2b57e17479e92e241234a241af7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c113c2b57e17479e92e241234a241af7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970500},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_c113c2b57e17479e92e241234a241af7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c113c2b57e17479e92e241234a241af7\",\"kudu.table_id\":\"da0cb25ccfed426ca61f0dd9f93d30ec\",\"transient_lastDdlTime\":\"1794970500\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f55fe6eed5d google::LogMessage::Fail() at ??:0
@ 0x7f55fe6f2b23 google::LogMessage::SendToLog() at ??:0
@ 0x7f55fe6ee75c google::LogMessage::Flush() at ??:0
I20261118 02:55:17.497677 4845 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:55:17.497782 4842 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_28c62f411da74545aa42dad7325bc064" } modify_external_catalogs: true
I20261118 02:55:17.498061 4842 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_28c62f411da74545aa42dad7325bc064" } modify_external_catalogs: true
@ 0x7f55fe6ee9a1 google::LogMessage::~LogMessage() at ??:0
I20261118 02:55:17.499245 4846 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet 1d347c0ae53241708ec623562f4740d4
I20261118 02:55:17.499305 4843 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:17.501046 4843 catalog_manager.cc:6936] ReplaceTablet: tablet a013d180f2cd430ab1ba30b39fd27591 deleted and replaced by tablet 8ae679d2feb04b95ab2f4fb51e443d64
I20261118 02:55:17.501396 4846 catalog_manager.cc:6936] ReplaceTablet: tablet 1d347c0ae53241708ec623562f4740d4 deleted and replaced by tablet a67bc60c7ecb4b7e8e03e04267bb91a5
I20261118 02:55:17.501461 4843 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42302:
name: "default.Table_e39f59e2fdbb47fda4c10beba29ad8f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:17.501746 4846 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_4f7796d5d5d54def843b971e66d15ea6" } new_table_name: "default.Table_79eb11797e3f4bc2bffe161096300f7a" modify_external_catalogs: true
@ 0x7f5613b6e9a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 02:55:17.508414 4845 catalog_manager.cc:6936] ReplaceTablet: tablet c92c29c30ab6411dabe6f02331d29b7a deleted and replaced by tablet b82ca8a282f94392bb2746ed7b8b8ba7
I20261118 02:55:17.509038 4845 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_a7f3710d39fb447da94e16d4ab3933fd" } modify_external_catalogs: true
I20261118 02:55:17.509253 4845 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "default.Table_a7f3710d39fb447da94e16d4ab3933fd" } modify_external_catalogs: true
@ 0x7f5613b6d251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20261118 02:55:17.524923 3734 tablet_service.cc:1547] Processing DeleteTablet for tablet c92c29c30ab6411dabe6f02331d29b7a with delete_type TABLET_DATA_DELETED (replaced by tablet b82ca8a282f94392bb2746ed7b8b8ba7) from {username='slave'} at 127.0.0.1:41072
@ 0x7f5613b74ecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20261118 02:55:17.528736 3733 tablet_service.cc:1547] Processing DeleteTablet for tablet a013d180f2cd430ab1ba30b39fd27591 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ae679d2feb04b95ab2f4fb51e443d64) from {username='slave'} at 127.0.0.1:41072
I20261118 02:55:17.529330 3733 tablet_service.cc:1547] Processing DeleteTablet for tablet 1d347c0ae53241708ec623562f4740d4 with delete_type TABLET_DATA_DELETED (replaced by tablet a67bc60c7ecb4b7e8e03e04267bb91a5) from {username='slave'} at 127.0.0.1:41072
I20261118 02:55:17.529987 3733 tablet_service.cc:1500] Processing CreateTablet for tablet 8ae679d2feb04b95ab2f4fb51e443d64 (DEFAULT_TABLE table=default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:17.530427 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae679d2feb04b95ab2f4fb51e443d64. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f5613b74d49 std::_Function_handler<>::_M_invoke() at ??:0
I20261118 02:55:17.530959 5075 tablet_replica.cc:333] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:17.531199 5075 raft_consensus.cc:2261] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
I20261118 02:55:17.531463 5075 raft_consensus.cc:2290] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:17.532120 5075 ts_tablet_manager.cc:1962] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f55ffb37dc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f56032126db start_thread at ??:0
I20261118 02:55:17.537914 3734 tablet_service.cc:1500] Processing CreateTablet for tablet a67bc60c7ecb4b7e8e03e04267bb91a5 (DEFAULT_TABLE table=default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:17.538292 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a67bc60c7ecb4b7e8e03e04267bb91a5. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f55fbc8261f clone at ??:0
I20261118 02:55:17.539463 5075 ts_tablet_manager.cc:1975] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:17.539587 5075 log.cc:1199] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:55:17.539908 5075 ts_tablet_manager.cc:1996] T c92c29c30ab6411dabe6f02331d29b7a P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:17.540879 5075 tablet_replica.cc:333] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:17.541091 5075 raft_consensus.cc:2261] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:17.541401 5075 raft_consensus.cc:2290] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:17.541858 5075 ts_tablet_manager.cc:1962] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:17.542951 3732 tablet_service.cc:1500] Processing CreateTablet for tablet b82ca8a282f94392bb2746ed7b8b8ba7 (DEFAULT_TABLE table=default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:17.543397 3732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b82ca8a282f94392bb2746ed7b8b8ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:17.545748 5077 tablet_bootstrap.cc:492] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:17.545882 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet a013d180f2cd430ab1ba30b39fd27591 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ae679d2feb04b95ab2f4fb51e443d64) from {username='slave'} at 127.0.0.1:36196
I20261118 02:55:17.546530 3877 tablet_service.cc:1500] Processing CreateTablet for tablet b82ca8a282f94392bb2746ed7b8b8ba7 (DEFAULT_TABLE table=default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:17.547045 3877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b82ca8a282f94392bb2746ed7b8b8ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:17.547571 5077 tablet_bootstrap.cc:654] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:17.548264 3879 tablet_service.cc:1547] Processing DeleteTablet for tablet c92c29c30ab6411dabe6f02331d29b7a with delete_type TABLET_DATA_DELETED (replaced by tablet b82ca8a282f94392bb2746ed7b8b8ba7) from {username='slave'} at 127.0.0.1:36196
I20261118 02:55:17.548484 3882 tablet_service.cc:1547] Processing DeleteTablet for tablet 1d347c0ae53241708ec623562f4740d4 with delete_type TABLET_DATA_DELETED (replaced by tablet a67bc60c7ecb4b7e8e03e04267bb91a5) from {username='slave'} at 127.0.0.1:36196
I20261118 02:55:17.548853 3880 tablet_service.cc:1500] Processing CreateTablet for tablet 8ae679d2feb04b95ab2f4fb51e443d64 (DEFAULT_TABLE table=default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:17.549240 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae679d2feb04b95ab2f4fb51e443d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:17.555918 3878 tablet_service.cc:1500] Processing CreateTablet for tablet a67bc60c7ecb4b7e8e03e04267bb91a5 (DEFAULT_TABLE table=default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:17.556412 3878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a67bc60c7ecb4b7e8e03e04267bb91a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:17.573114 5075 ts_tablet_manager.cc:1975] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:17.573272 5075 log.cc:1199] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:17.573688 5075 ts_tablet_manager.cc:1996] T a013d180f2cd430ab1ba30b39fd27591 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:17.574684 5075 tablet_replica.cc:333] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:17.574885 5075 raft_consensus.cc:2261] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:17.575067 5075 raft_consensus.cc:2290] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:17.575676 5075 ts_tablet_manager.cc:1962] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:17.577378 5080 tablet_bootstrap.cc:492] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:17.579586 5080 tablet_bootstrap.cc:654] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:17.587100 5075 ts_tablet_manager.cc:1975] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:17.587204 5075 log.cc:1199] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/1d347c0ae53241708ec623562f4740d4
I20261118 02:55:17.587567 5075 ts_tablet_manager.cc:1996] T 1d347c0ae53241708ec623562f4740d4 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:17.589464 5080 tablet_bootstrap.cc:492] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:17.589607 5080 ts_tablet_manager.cc:1449] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20261118 02:55:17.590394 5080 raft_consensus.cc:360] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.590595 5080 raft_consensus.cc:386] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:17.590677 5080 raft_consensus.cc:748] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:17.591002 5080 consensus_queue.cc:261] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.592978 5078 tablet_replica.cc:333] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:17.593180 5078 raft_consensus.cc:2261] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:17.593397 5078 raft_consensus.cc:2290] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:17.593942 5078 ts_tablet_manager.cc:1962] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:17.597049 5080 ts_tablet_manager.cc:1480] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20261118 02:55:17.600358 5078 ts_tablet_manager.cc:1975] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:17.600457 5078 log.cc:1199] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:17.600832 5078 ts_tablet_manager.cc:1996] T a013d180f2cd430ab1ba30b39fd27591 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:17.601270 5080 tablet_bootstrap.cc:492] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:17.601788 5078 tablet_replica.cc:333] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:17.602504 5078 raft_consensus.cc:2261] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:17.602813 5078 raft_consensus.cc:2290] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:17.603102 5080 tablet_bootstrap.cc:654] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:17.603363 5078 ts_tablet_manager.cc:1962] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:17.607169 5080 tablet_bootstrap.cc:492] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:17.607319 5080 ts_tablet_manager.cc:1449] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20261118 02:55:17.607362 5077 tablet_bootstrap.cc:492] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:17.607503 5077 ts_tablet_manager.cc:1449] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.046s
I20261118 02:55:17.608132 5080 raft_consensus.cc:360] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.608337 5080 raft_consensus.cc:386] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:17.608412 5080 raft_consensus.cc:748] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:17.608515 5077 raft_consensus.cc:360] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.608704 5077 raft_consensus.cc:386] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:17.608784 5077 raft_consensus.cc:748] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:17.608922 5080 consensus_queue.cc:261] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.609016 5077 consensus_queue.cc:261] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.609579 5077 ts_tablet_manager.cc:1480] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261118 02:55:17.609869 5077 tablet_bootstrap.cc:492] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:17.610651 5078 ts_tablet_manager.cc:1975] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:17.610762 5078 log.cc:1199] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:55:17.611131 5078 ts_tablet_manager.cc:1996] T c92c29c30ab6411dabe6f02331d29b7a P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:17.611649 5077 tablet_bootstrap.cc:654] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:17.611940 5080 ts_tablet_manager.cc:1480] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20261118 02:55:17.612023 5078 tablet_replica.cc:333] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:17.612258 5078 raft_consensus.cc:2261] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:17.612262 5080 tablet_bootstrap.cc:492] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:17.612458 5078 raft_consensus.cc:2290] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:17.612984 5078 ts_tablet_manager.cc:1962] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:17.614028 5080 tablet_bootstrap.cc:654] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:17.619131 5078 ts_tablet_manager.cc:1975] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:17.619253 5078 log.cc:1199] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/1d347c0ae53241708ec623562f4740d4
I20261118 02:55:17.619630 5078 ts_tablet_manager.cc:1996] T 1d347c0ae53241708ec623562f4740d4 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:17.620595 5077 tablet_bootstrap.cc:492] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:17.621766 5077 ts_tablet_manager.cc:1449] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20261118 02:55:17.622485 5077 raft_consensus.cc:360] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.622645 5080 tablet_bootstrap.cc:492] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:17.622658 5077 raft_consensus.cc:386] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:17.622720 5077 raft_consensus.cc:748] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:17.622758 5080 ts_tablet_manager.cc:1449] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20261118 02:55:17.622977 5077 consensus_queue.cc:261] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.623397 5080 raft_consensus.cc:360] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.623488 5077 ts_tablet_manager.cc:1480] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20261118 02:55:17.623556 5080 raft_consensus.cc:386] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:17.623627 5080 raft_consensus.cc:748] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:17.623852 5080 consensus_queue.cc:261] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.626335 5080 ts_tablet_manager.cc:1480] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20261118 02:55:17.627665 4612 raft_consensus.cc:499] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:17.627800 4612 raft_consensus.cc:521] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.628818 4612 leader_election.cc:290] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:17.629498 5077 tablet_bootstrap.cc:492] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:17.629948 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a67bc60c7ecb4b7e8e03e04267bb91a5" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
W20261118 02:55:17.629972 3691 leader_election.cc:343] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Not found: Tablet not found: a67bc60c7ecb4b7e8e03e04267bb91a5
I20261118 02:55:17.630139 3901 raft_consensus.cc:2490] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:17.630614 3690 leader_election.cc:304] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:17.630906 4612 raft_consensus.cc:2826] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:17.631014 4612 raft_consensus.cc:499] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:17.631106 4612 raft_consensus.cc:3096] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:17.631256 5077 tablet_bootstrap.cc:654] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:17.634172 4612 raft_consensus.cc:521] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.634842 5077 tablet_bootstrap.cc:492] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:17.634900 4612 leader_election.cc:290] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:17.635076 5077 ts_tablet_manager.cc:1449] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20261118 02:55:17.635357 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "a67bc60c7ecb4b7e8e03e04267bb91a5" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:17.635545 3901 raft_consensus.cc:3096] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:17.635860 5077 raft_consensus.cc:360] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.636040 5077 raft_consensus.cc:386] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20261118 02:55:17.635906 3691 leader_election.cc:343] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Not found: Tablet not found: a67bc60c7ecb4b7e8e03e04267bb91a5
I20261118 02:55:17.636250 5077 raft_consensus.cc:748] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:17.636503 5077 consensus_queue.cc:261] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.637009 5077 ts_tablet_manager.cc:1480] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20261118 02:55:17.638312 3901 raft_consensus.cc:2490] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:17.638796 3690 leader_election.cc:304] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:17.639231 4612 raft_consensus.cc:2826] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:17.639664 4612 raft_consensus.cc:705] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:17.639998 4612 consensus_queue.cc:238] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.642830 5090 consensus_queue.cc:1049] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 02:55:17.643293 3691 consensus_peers.cc:597] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a67bc60c7ecb4b7e8e03e04267bb91a5. This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:17.740816 4612 raft_consensus.cc:499] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:17.741046 4612 raft_consensus.cc:521] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.741828 4612 leader_election.cc:290] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:17.742376 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b82ca8a282f94392bb2746ed7b8b8ba7" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:17.742729 3901 raft_consensus.cc:2490] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
W20261118 02:55:17.742722 3691 leader_election.cc:343] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7
I20261118 02:55:17.743161 3690 leader_election.cc:304] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:17.743471 4612 raft_consensus.cc:2826] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:17.743564 4612 raft_consensus.cc:499] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:17.743664 4612 raft_consensus.cc:3096] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:17.745707 4612 raft_consensus.cc:521] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.746248 4612 leader_election.cc:290] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:17.746776 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b82ca8a282f94392bb2746ed7b8b8ba7" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:17.746963 3901 raft_consensus.cc:3096] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
W20261118 02:55:17.747074 3691 leader_election.cc:343] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7
I20261118 02:55:17.749910 3901 raft_consensus.cc:2490] T b82ca8a282f94392bb2746ed7b8b8ba7 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:17.750299 3690 leader_election.cc:304] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:17.750545 4612 raft_consensus.cc:2826] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:17.750725 4612 raft_consensus.cc:705] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:17.750998 4612 consensus_queue.cc:238] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:17.754232 4612 consensus_queue.cc:1049] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 02:55:17.754529 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 1: this message will repeat every 5th retry.
W20261118 02:55:17.777125 3690 connection.cc:573] server connection from 127.0.0.1:41072 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:17.777125 3835 connection.cc:573] server connection from 127.0.0.1:36196 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:17.777477 3836 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:17.777351 5069 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:17.504693 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.2.229.65:33705)
1118 02:55:17.530605 (+ 25912us) server_negotiation.cc:207] Beginning negotiation
1118 02:55:17.530611 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1118 02:55:17.530630 (+ 19us) server_negotiation.cc:408] Connection header received
1118 02:55:17.530705 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1118 02:55:17.530710 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1118 02:55:17.530781 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1118 02:55:17.531519 (+ 738us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1118 02:55:17.531760 (+ 241us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1118 02:55:17.533236 (+ 1476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1118 02:55:17.777302 (+244066us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46584: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":25783,"thread_start_us":25679,"threads_started":1}
W20261118 02:55:17.778443 3689 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:17.779369 3955 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:17.780386 3955 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.229.125:38339)
W20261118 02:55:17.780925 3955 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.229.125:38339)
W20261118 02:55:17.781210 3955 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.229.125:38339)
W20261118 02:55:17.781391 3955 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.229.125:38339)
W20261118 02:55:17.880867 3553 consensus_peers.cc:597] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 -> Peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Couldn't send request to peer 3f1973986ec1466eab5bec01dd81654f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d347c0ae53241708ec623562f4740d4. This is attempt 1: this message will repeat every 5th retry.
W20261118 02:55:17.976025 3552 consensus_peers.cc:597] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 -> Peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Couldn't send request to peer 99a39b1d9d814deeb3a8a644e9b392ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d347c0ae53241708ec623562f4740d4. This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:18.019960 5100 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:18.020197 5100 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:18.021987 5100 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:18.022921 4970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:18.024148 4969 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:18.024335 4970 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:18.024495 4970 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:18.024865 5100 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 02:55:18.190639 4612 raft_consensus.cc:499] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:18.190848 4612 raft_consensus.cc:521] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:18.191565 4612 leader_election.cc:290] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:18.192157 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8ae679d2feb04b95ab2f4fb51e443d64" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:18.192366 3902 raft_consensus.cc:2490] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
W20261118 02:55:18.192399 3691 leader_election.cc:343] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64
I20261118 02:55:18.192745 3690 leader_election.cc:304] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:18.193009 4612 raft_consensus.cc:2826] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:18.193091 4612 raft_consensus.cc:499] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:18.193192 4612 raft_consensus.cc:3096] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:18.195111 4612 raft_consensus.cc:521] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:18.195698 4612 leader_election.cc:290] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:18.196218 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8ae679d2feb04b95ab2f4fb51e443d64" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:18.196403 3902 raft_consensus.cc:3096] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
W20261118 02:55:18.196491 3691 leader_election.cc:343] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64
I20261118 02:55:18.198383 3902 raft_consensus.cc:2490] T 8ae679d2feb04b95ab2f4fb51e443d64 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:18.198832 3690 leader_election.cc:304] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:18.199155 4612 raft_consensus.cc:2826] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:18.199343 4612 raft_consensus.cc:705] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:18.199625 4612 consensus_queue.cc:238] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:18.202332 4612 consensus_queue.cc:1049] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 02:55:18.202572 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:18.514424 5108 ts_tablet_manager.cc:979] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Initiating tablet copy from peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:18.515405 5108 tablet_copy_client.cc:323] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.229.66:40377
I20261118 02:55:18.516284 3775 tablet_copy_service.cc:140] P 99a39b1d9d814deeb3a8a644e9b392ef: Received BeginTabletCopySession request for tablet b82ca8a282f94392bb2746ed7b8b8ba7 from peer 1fdf7d7ece324fefb3a0c376add5b7e8 ({username='slave'} at 127.2.229.65:55185)
I20261118 02:55:18.516482 3775 tablet_copy_service.cc:161] P 99a39b1d9d814deeb3a8a644e9b392ef: Beginning new tablet copy session on tablet b82ca8a282f94392bb2746ed7b8b8ba7 from peer 1fdf7d7ece324fefb3a0c376add5b7e8 at {username='slave'} at 127.2.229.65:55185: session id = 1fdf7d7ece324fefb3a0c376add5b7e8-b82ca8a282f94392bb2746ed7b8b8ba7
I20261118 02:55:18.518481 3775 tablet_copy_source_session.cc:215] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef: Tablet Copy: opened 0 blocks and 1 log segments
I20261118 02:55:18.520277 5108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b82ca8a282f94392bb2746ed7b8b8ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:18.525393 5108 tablet_copy_client.cc:806] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Starting download of 0 data blocks...
I20261118 02:55:18.525678 5108 tablet_copy_client.cc:670] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Starting download of 1 WAL segments...
I20261118 02:55:18.527612 5108 tablet_copy_client.cc:538] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261118 02:55:18.530397 5108 tablet_bootstrap.cc:492] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:18.530742 5110 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:18.530900 5110 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:18.531637 5110 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:18.533679 4970 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:18.534319 4969 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:18.534439 4969 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:18.534683 5110 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 02:55:18.536306 5108 tablet_bootstrap.cc:492] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:18.536657 5108 tablet_bootstrap.cc:492] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap complete.
I20261118 02:55:18.536934 5108 ts_tablet_manager.cc:1449] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.004s
I20261118 02:55:18.537604 5108 raft_consensus.cc:360] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:18.537762 5108 raft_consensus.cc:748] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:18.538019 5108 consensus_queue.cc:261] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:18.539575 5108 ts_tablet_manager.cc:1480] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 02:55:18.540712 3775 tablet_copy_service.cc:342] P 99a39b1d9d814deeb3a8a644e9b392ef: Request end of tablet copy session 1fdf7d7ece324fefb3a0c376add5b7e8-b82ca8a282f94392bb2746ed7b8b8ba7 received from {username='slave'} at 127.2.229.65:55185
I20261118 02:55:18.540854 3775 tablet_copy_service.cc:434] P 99a39b1d9d814deeb3a8a644e9b392ef: ending tablet copy session 1fdf7d7ece324fefb3a0c376add5b7e8-b82ca8a282f94392bb2746ed7b8b8ba7 on tablet b82ca8a282f94392bb2746ed7b8b8ba7 with peer 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:18.781210 5108 ts_tablet_manager.cc:979] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Initiating tablet copy from peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:18.781842 5108 tablet_copy_client.cc:323] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.229.66:40377
I20261118 02:55:18.782650 3775 tablet_copy_service.cc:140] P 99a39b1d9d814deeb3a8a644e9b392ef: Received BeginTabletCopySession request for tablet a67bc60c7ecb4b7e8e03e04267bb91a5 from peer 1fdf7d7ece324fefb3a0c376add5b7e8 ({username='slave'} at 127.2.229.65:55185)
I20261118 02:55:18.782800 3775 tablet_copy_service.cc:161] P 99a39b1d9d814deeb3a8a644e9b392ef: Beginning new tablet copy session on tablet a67bc60c7ecb4b7e8e03e04267bb91a5 from peer 1fdf7d7ece324fefb3a0c376add5b7e8 at {username='slave'} at 127.2.229.65:55185: session id = 1fdf7d7ece324fefb3a0c376add5b7e8-a67bc60c7ecb4b7e8e03e04267bb91a5
I20261118 02:55:18.784861 3775 tablet_copy_source_session.cc:215] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 99a39b1d9d814deeb3a8a644e9b392ef: Tablet Copy: opened 0 blocks and 1 log segments
I20261118 02:55:18.786069 5108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a67bc60c7ecb4b7e8e03e04267bb91a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:18.790584 5108 tablet_copy_client.cc:806] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Starting download of 0 data blocks...
I20261118 02:55:18.790864 5108 tablet_copy_client.cc:670] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Starting download of 1 WAL segments...
I20261118 02:55:18.792666 5108 tablet_copy_client.cc:538] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261118 02:55:18.795193 5108 tablet_bootstrap.cc:492] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:18.799563 5108 tablet_bootstrap.cc:492] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:18.799887 5108 tablet_bootstrap.cc:492] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap complete.
I20261118 02:55:18.800124 5108 ts_tablet_manager.cc:1449] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20261118 02:55:18.800774 5108 raft_consensus.cc:360] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:18.800941 5108 raft_consensus.cc:748] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:18.801184 5108 consensus_queue.cc:261] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:18.801667 5108 ts_tablet_manager.cc:1480] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20261118 02:55:18.802863 3775 tablet_copy_service.cc:342] P 99a39b1d9d814deeb3a8a644e9b392ef: Request end of tablet copy session 1fdf7d7ece324fefb3a0c376add5b7e8-a67bc60c7ecb4b7e8e03e04267bb91a5 received from {username='slave'} at 127.2.229.65:55185
I20261118 02:55:18.803018 3775 tablet_copy_service.cc:434] P 99a39b1d9d814deeb3a8a644e9b392ef: ending tablet copy session 1fdf7d7ece324fefb3a0c376add5b7e8-a67bc60c7ecb4b7e8e03e04267bb91a5 on tablet a67bc60c7ecb4b7e8e03e04267bb91a5 with peer 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:19.165515 5108 ts_tablet_manager.cc:979] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: Initiating tablet copy from peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:19.166119 5108 tablet_copy_client.cc:323] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.229.66:40377
I20261118 02:55:19.166812 3775 tablet_copy_service.cc:140] P 99a39b1d9d814deeb3a8a644e9b392ef: Received BeginTabletCopySession request for tablet 8ae679d2feb04b95ab2f4fb51e443d64 from peer 1fdf7d7ece324fefb3a0c376add5b7e8 ({username='slave'} at 127.2.229.65:55185)
I20261118 02:55:19.166988 3775 tablet_copy_service.cc:161] P 99a39b1d9d814deeb3a8a644e9b392ef: Beginning new tablet copy session on tablet 8ae679d2feb04b95ab2f4fb51e443d64 from peer 1fdf7d7ece324fefb3a0c376add5b7e8 at {username='slave'} at 127.2.229.65:55185: session id = 1fdf7d7ece324fefb3a0c376add5b7e8-8ae679d2feb04b95ab2f4fb51e443d64
I20261118 02:55:19.169787 3775 tablet_copy_source_session.cc:215] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef: Tablet Copy: opened 0 blocks and 1 log segments
I20261118 02:55:19.171038 5108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae679d2feb04b95ab2f4fb51e443d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:19.175735 5108 tablet_copy_client.cc:806] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Starting download of 0 data blocks...
I20261118 02:55:19.176014 5108 tablet_copy_client.cc:670] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Starting download of 1 WAL segments...
I20261118 02:55:19.177559 5108 tablet_copy_client.cc:538] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261118 02:55:19.179965 5108 tablet_bootstrap.cc:492] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:19.183944 5108 tablet_bootstrap.cc:492] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:19.184286 5108 tablet_bootstrap.cc:492] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap complete.
I20261118 02:55:19.184455 5108 ts_tablet_manager.cc:1449] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.001s
I20261118 02:55:19.185002 5108 raft_consensus.cc:360] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:19.185158 5108 raft_consensus.cc:748] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:19.185338 5108 consensus_queue.cc:261] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:19.185760 5108 ts_tablet_manager.cc:1480] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20261118 02:55:19.187186 3775 tablet_copy_service.cc:342] P 99a39b1d9d814deeb3a8a644e9b392ef: Request end of tablet copy session 1fdf7d7ece324fefb3a0c376add5b7e8-8ae679d2feb04b95ab2f4fb51e443d64 received from {username='slave'} at 127.2.229.65:55185
I20261118 02:55:19.187378 3775 tablet_copy_service.cc:434] P 99a39b1d9d814deeb3a8a644e9b392ef: ending tablet copy session 1fdf7d7ece324fefb3a0c376add5b7e8-8ae679d2feb04b95ab2f4fb51e443d64 on tablet 8ae679d2feb04b95ab2f4fb51e443d64 with peer 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:19.260597 5132 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:19.260775 5132 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:19.261348 5132 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:19.263741 4969 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:19.263900 4970 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:19.264035 4970 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:19.264321 5132 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 02:55:19.728300 3616 raft_consensus.cc:1227] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20261118 02:55:19.918701 3616 raft_consensus.cc:1227] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20261118 02:55:19.946965 3616 raft_consensus.cc:1227] T a67bc60c7ecb4b7e8e03e04267bb91a5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20261118 02:55:19.960668 5144 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:55:19.960837 5144 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:19.961663 5144 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:19.962544 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:55:19.962762 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:55:19.962883 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:19.963148 5144 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:20.042069 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.126:38025
--webserver_interface=127.2.229.126
--webserver_port=44955
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 02:55:20.120795 5143 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:55:20.120985 5143 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:20.121806 5143 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:20.122754 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:55:20.122915 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:20.123008 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:20.123252 5143 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:55:20.398454 5146 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:55:20.398861 5146 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:55:20.398989 5146 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:55:20.407805 5146 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:55:20.407904 5146 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:55:20.407989 5146 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:55:20.408149 5146 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:55:20.420967 5146 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.126:38025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.229.126
--webserver_port=44955
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:55:20.422909 5146 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:55:20.424867 5146 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:55:20.433331 5153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:20.433562 5154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:20.434264 5156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:55:20.435442 5146 server_base.cc:1061] running on GCE node
I20261118 02:55:20.436265 5146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:55:20.437827 5146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:55:20.439064 5146 hybrid_clock.cc:648] HybridClock initialized: now 1794970520438992 us; error 49 us; skew 500 ppm
I20261118 02:55:20.439438 5146 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:55:20.444727 5146 webserver.cc:577] Webserver started at http://127.2.229.126:44955/ using document root <none> and password file <none>
I20261118 02:55:20.445528 5146 fs_manager.cc:362] Metadata directory not provided
I20261118 02:55:20.445631 5146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:55:20.452905 5146 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20261118 02:55:20.456629 5164 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:55:20.458259 5146 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20261118 02:55:20.458413 5146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
uuid: "26c6b20fa203404a88463ea27bdd3c7d"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:55:20.459086 5146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:55:20.504082 5146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:55:20.505003 5146 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:55:20.505275 5146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:55:20.525074 5217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.126:38025 every 8 connection(s)
I20261118 02:55:20.525087 5146 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.126:38025
I20261118 02:55:20.526829 5146 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-0/data/info.pb
I20261118 02:55:20.533906 5218 sys_catalog.cc:263] Verifying existing consensus state
I20261118 02:55:20.534224 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 5146
I20261118 02:55:20.538407 5218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Bootstrap starting.
I20261118 02:55:20.564605 5218 log.cc:826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:20.637187 3663 heartbeater.cc:345] Connected to a master server at 127.2.229.126:38025
I20261118 02:55:20.637392 3663 heartbeater.cc:500] Master 127.2.229.126:38025 was elected leader, sending a full tablet report...
I20261118 02:55:20.644713 5218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=101 overwritten=0 applied=101 ignored=0} inserts{seen=67 ignored=0} mutations{seen=87 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:20.645326 5218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Bootstrap complete.
I20261118 02:55:20.650624 5218 raft_consensus.cc:360] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:20.651289 5218 raft_consensus.cc:748] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c6b20fa203404a88463ea27bdd3c7d, State: Initialized, Role: FOLLOWER
I20261118 02:55:20.651968 5218 consensus_queue.cc:261] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 101, Last appended: 6.101, Last appended by leader: 101, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:20.652774 5225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:20.653012 5225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:20.653786 5218 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:55:20.666072 5239 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:20.666260 5239 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: loading cluster ID for follower catalog manager: success
I20261118 02:55:20.671509 5239 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: acquiring CA information for follower catalog manager: success
I20261118 02:55:20.674789 5239 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:55:20.685365 5146 master_runner.cc:187] Error getting master registration for 127.2.229.125:38339: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111),
I20261118 02:55:20.689010 5146 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:20.720813 3947 heartbeater.cc:345] Connected to a master server at 127.2.229.126:38025
I20261118 02:55:20.723443 5183 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" instance_seqno: 1794970499527172) as {username='slave'} at 127.2.229.67:39849; Asking this server to re-register.
I20261118 02:55:20.724186 3947 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:20.726233 5183 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:20.740029 5245 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:20.740310 5245 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:20.741003 5245 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:20.743934 4969 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:20.748364 5193 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 106 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:55:20.748924 5193 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 813a6937cea442ef81645fccc32207f4 in term 6.
I20261118 02:55:20.749716 4970 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4; no voters: 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:20.750082 5245 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20261118 02:55:20.750213 5245 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:20.750319 5245 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 7 FOLLOWER]: Advancing to term 8
I20261118 02:55:20.753501 5245 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:20.754292 5245 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 election: Requested vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
I20261118 02:55:20.754891 5193 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "813a6937cea442ef81645fccc32207f4" candidate_term: 8 candidate_status { last_received { term: 7 index: 106 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d"
I20261118 02:55:20.755136 5193 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 6 FOLLOWER]: Advancing to term 8
W20261118 02:55:20.755834 4969 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:20.759920 5193 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 813a6937cea442ef81645fccc32207f4 in term 8.
I20261118 02:55:20.760392 4970 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4; no voters: 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:20.760735 5245 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 8 FOLLOWER]: Leader election won for term 8
I20261118 02:55:20.761154 5245 raft_consensus.cc:705] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 8 LEADER]: Becoming Leader. State: Replica: 813a6937cea442ef81645fccc32207f4, State: Running, Role: LEADER
I20261118 02:55:20.761704 5245 consensus_queue.cc:238] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 7.106, Last appended by leader: 106, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:20.763628 5249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 813a6937cea442ef81645fccc32207f4. Latest consensus state: current_term: 8 leader_uuid: "813a6937cea442ef81645fccc32207f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:20.763988 5249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: LEADER
W20261118 02:55:20.766536 4969 consensus_peers.cc:597] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 -> Peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Couldn't send request to peer 457eb38cba564ecd9f6f11245172260b. Status: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:20.767616 5193 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Refusing update from remote peer 813a6937cea442ef81645fccc32207f4: Log matching property violated. Preceding OpId in replica: term: 6 index: 101. Preceding OpId from leader: term: 8 index: 107. (index mismatch)
I20261118 02:55:20.768399 5245 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 101, Time since last communication: 0.000s
I20261118 02:55:20.769163 5252 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 02:55:20.771977 5252 catalog_manager.cc:694] Loaded metadata for table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190]
I20261118 02:55:20.772516 5252 catalog_manager.cc:694] Loaded metadata for table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c]
I20261118 02:55:20.773025 5252 catalog_manager.cc:694] Loaded metadata for table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]
I20261118 02:55:20.773561 5252 catalog_manager.cc:694] Loaded metadata for table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]
I20261118 02:55:20.774039 5252 catalog_manager.cc:694] Loaded metadata for table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]
I20261118 02:55:20.774552 5252 catalog_manager.cc:694] Loaded metadata for table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]
I20261118 02:55:20.775027 5252 catalog_manager.cc:694] Loaded metadata for table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da]
I20261118 02:55:20.775450 5252 catalog_manager.cc:694] Loaded metadata for table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]
I20261118 02:55:20.775779 5252 catalog_manager.cc:694] Loaded metadata for table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d]
I20261118 02:55:20.776177 5252 catalog_manager.cc:694] Loaded metadata for table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]
I20261118 02:55:20.776612 5252 catalog_manager.cc:694] Loaded metadata for table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]
I20261118 02:55:20.777097 5252 catalog_manager.cc:694] Loaded metadata for table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b]
I20261118 02:55:20.777577 5252 catalog_manager.cc:694] Loaded metadata for table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]
I20261118 02:55:20.778100 5252 catalog_manager.cc:694] Loaded metadata for table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]
I20261118 02:55:20.778455 5252 catalog_manager.cc:694] Loaded metadata for table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]
I20261118 02:55:20.778784 5252 catalog_manager.cc:694] Loaded metadata for table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]
I20261118 02:55:20.779155 5252 catalog_manager.cc:694] Loaded metadata for table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]
I20261118 02:55:20.779510 5252 catalog_manager.cc:694] Loaded metadata for table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1]
I20261118 02:55:20.779874 5252 catalog_manager.cc:694] Loaded metadata for table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f]
I20261118 02:55:20.780372 5252 catalog_manager.cc:694] Loaded metadata for table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]
I20261118 02:55:20.783121 5252 tablet_loader.cc:96] loaded metadata for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7])
I20261118 02:55:20.783512 5252 tablet_loader.cc:96] loaded metadata for tablet 0e901a03b90e4c978eb98be87163cfe5 (table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190])
I20261118 02:55:20.783813 5252 tablet_loader.cc:96] loaded metadata for tablet 1904140da78b4567a9decb250c6f74a3 (table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f])
I20261118 02:55:20.784343 5252 tablet_loader.cc:96] loaded metadata for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db])
I20261118 02:55:20.784638 5252 tablet_loader.cc:96] loaded metadata for tablet 2c92955aa65142f1bd20f4b8f3321500 (table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d])
I20261118 02:55:20.785084 5252 tablet_loader.cc:96] loaded metadata for tablet 46f05790dbdc4b8185a7b0362f604cf8 (table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf])
I20261118 02:55:20.786100 5252 tablet_loader.cc:96] loaded metadata for tablet 65bd6fcaed5b47c3a2478d375ace9636 (table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73])
I20261118 02:55:20.786552 5252 tablet_loader.cc:96] loaded metadata for tablet 72f427c72a554ac1a0a2f2c3a78865da (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b])
I20261118 02:55:20.786835 5252 tablet_loader.cc:96] loaded metadata for tablet 78189cccd457466bb5d01231038fec24 (table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da])
I20261118 02:55:20.787108 5252 tablet_loader.cc:96] loaded metadata for tablet 8ae679d2feb04b95ab2f4fb51e443d64 (table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d])
I20261118 02:55:20.787382 5252 tablet_loader.cc:96] loaded metadata for tablet 8bed657d0da64d95851f8a86216b6c2c (table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c])
I20261118 02:55:20.787925 5252 tablet_loader.cc:96] loaded metadata for tablet 973831392ec24e0681c3d4fffd8bb199 (table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139])
I20261118 02:55:20.789000 5252 tablet_loader.cc:96] loaded metadata for tablet a67bc60c7ecb4b7e8e03e04267bb91a5 (table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b])
I20261118 02:55:20.789693 5252 tablet_loader.cc:96] loaded metadata for tablet ab61b43e639a4e75bc2e7009ea86a64f (table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67])
I20261118 02:55:20.790086 5252 tablet_loader.cc:96] loaded metadata for tablet b129518ce23c45758ebcaa55410757d8 (table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1])
I20261118 02:55:20.790717 5252 tablet_loader.cc:96] loaded metadata for tablet b82ca8a282f94392bb2746ed7b8b8ba7 (table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d])
I20261118 02:55:20.790992 5252 tablet_loader.cc:96] loaded metadata for tablet c0713dcf79f54873a99d7556ad8f1454 (table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec])
I20261118 02:55:20.791435 5252 tablet_loader.cc:96] loaded metadata for tablet ccf78258c9634540a178855a142f4d4b (table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e])
I20261118 02:55:20.791854 5252 tablet_loader.cc:96] loaded metadata for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6])
I20261118 02:55:20.792527 5252 tablet_loader.cc:96] loaded metadata for tablet ffe8d4c111d14054b76f9c7724072a94 (table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66])
I20261118 02:55:20.792847 5252 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 02:55:20.794230 5252 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:20.794315 5252 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:55:20.796497 5225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 813a6937cea442ef81645fccc32207f4. Latest consensus state: current_term: 8 leader_uuid: "813a6937cea442ef81645fccc32207f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:20.796767 5225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:20.797349 5252 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:55:20.797784 5254 mvcc.cc:204] Tried to move back new op lower bound from 7352199239705354240 to 7352199238828982272. Current Snapshot: MvccSnapshot[applied={T|T < 7352199239705354240}]
I20261118 02:55:20.799089 5252 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Loaded TSK: 0
I20261118 02:55:20.799180 5256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "813a6937cea442ef81645fccc32207f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:20.799484 5256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:20.800175 5252 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 02:55:20.800987 5256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "813a6937cea442ef81645fccc32207f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:20.801216 5256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:20.802155 5253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "813a6937cea442ef81645fccc32207f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:20.802385 5252 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:55:20.802462 5253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:20.803807 5252 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: 20
I20261118 02:55:20.817159 4982 master_service.cc:965] ReplaceTablet: received request to replace tablet 8ae679d2feb04b95ab2f4fb51e443d64 from {username='slave'} at 127.0.0.1:37916
I20261118 02:55:20.818449 4976 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37916:
table { table_name: "default.Table_287099a68a3540f1be0284c71d4d7056" } modify_external_catalogs: true
I20261118 02:55:20.818728 4976 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37916:
table { table_name: "default.Table_287099a68a3540f1be0284c71d4d7056" } modify_external_catalogs: true
I20261118 02:55:20.819249 4975 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37916:
name: "default.Table_51d36ff687ca47b5995e5693d7b8931b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:20.820135 4981 master_service.cc:965] ReplaceTablet: received request to replace tablet c0713dcf79f54873a99d7556ad8f1454 from {username='slave'} at 127.0.0.1:37916
I20261118 02:55:20.820236 4977 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37916:
name: "default.Table_8139578347f041afbe51671d3be78447"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:20.820909 4979 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37916:
name: "default.Table_ee3815d9524243ec886bd171ccaff9e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:20.821393 4980 master_service.cc:965] ReplaceTablet: received request to replace tablet 973831392ec24e0681c3d4fffd8bb199 from {username='slave'} at 127.0.0.1:37916
I20261118 02:55:20.822626 4983 master_service.cc:965] ReplaceTablet: received request to replace tablet ffe8d4c111d14054b76f9c7724072a94 from {username='slave'} at 127.0.0.1:37916
I20261118 02:55:20.823712 4984 master_service.cc:965] ReplaceTablet: received request to replace tablet b82ca8a282f94392bb2746ed7b8b8ba7 from {username='slave'} at 127.0.0.1:37916
I20261118 02:55:20.819316 4978 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37916:
name: "default.Table_1d3bb660e14d4101996960be8caa8f81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20261118 02:55:20.831734 5036 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 21, previous 21; dumping first 2 out of 15 received events: NotificationEvent(eventId=21, eventTime=1794970503, eventType=CREATE_TABLE, dbName=default, tableName=table_f0ae490a0df64e1097e144c7a370df61, message={"server":"","servicePrincipal":"","db":"default","table":"table_f0ae490a0df64e1097e144c7a370df61","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f0ae490a0df64e1097e144c7a370df61\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970503},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_f0ae490a0df64e1097e144c7a370df61\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f0ae490a0df64e1097e144c7a370df61\",\"kudu.table_id\":\"a59b9425c01049269883cf7767936c67\",\"transient_lastDdlTime\":\"1794970503\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1794970503, eventType=DROP_TABLE, dbName=default, tableName=table_c113c2b57e17479e92e241234a241af7, message={"server":"","servicePrincipal":"","db":"default","table":"table_c113c2b57e17479e92e241234a241af7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c113c2b57e17479e92e241234a241af7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970500},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_c113c2b57e17479e92e241234a241af7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c113c2b57e17479e92e241234a241af7\",\"kudu.table_id\":\"da0cb25ccfed426ca61f0dd9f93d30ec\",\"transient_lastDdlTime\":\"1794970500\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fcb6afd3d5d google::LogMessage::Fail() at ??:0
@ 0x7fcb6afd7b23 google::LogMessage::SendToLog() at ??:0
@ 0x7fcb6afd375c google::LogMessage::Flush() at ??:0
@ 0x7fcb6afd39a1 google::LogMessage::~LogMessage() at ??:0
I20261118 02:55:20.849900 3663 heartbeater.cc:500] Master 127.2.229.126:38025 was elected leader, sending a full tablet report...
I20261118 02:55:20.854040 5183 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" instance_seqno: 1794970498532688) as {username='slave'} at 127.2.229.65:43605; Asking this server to re-register.
I20261118 02:55:20.856480 3663 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:20.858747 5183 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:20.868574 4981 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Sending DeleteTablet for 3 replicas of tablet c0713dcf79f54873a99d7556ad8f1454
@ 0x7fcb804539a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 02:55:20.870682 4981 catalog_manager.cc:6936] ReplaceTablet: tablet c0713dcf79f54873a99d7556ad8f1454 deleted and replaced by tablet fbdc4c7b574e4f94b816fb0c0b14c185
@ 0x7fcb80452251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20261118 02:55:20.880270 4980 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Sending DeleteTablet for 3 replicas of tablet 973831392ec24e0681c3d4fffd8bb199
I20261118 02:55:20.880832 4980 catalog_manager.cc:6936] ReplaceTablet: tablet 973831392ec24e0681c3d4fffd8bb199 deleted and replaced by tablet 462a9be0ae114b01a9c69416185803f8
@ 0x7fcb80459ecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20261118 02:55:20.893810 4980 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37916:
table { table_name: "default.Table_0385cef6593c419eb426657c8cbef2bf" } new_table_name: "default.Table_496441d99faa4e2b8c7cd98e4a5ce745" modify_external_catalogs: true
I20261118 02:55:20.897856 4981 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37916:
name: "default.Table_dc982811b980416484416db1f74c685d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:20.899350 4983 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Sending DeleteTablet for 3 replicas of tablet ffe8d4c111d14054b76f9c7724072a94
I20261118 02:55:20.899824 4983 catalog_manager.cc:6936] ReplaceTablet: tablet ffe8d4c111d14054b76f9c7724072a94 deleted and replaced by tablet b119ead4bb6548b4a37da02ed6daa6a8
I20261118 02:55:20.900300 4984 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Sending DeleteTablet for 3 replicas of tablet b82ca8a282f94392bb2746ed7b8b8ba7
I20261118 02:55:20.900713 4984 catalog_manager.cc:6936] ReplaceTablet: tablet b82ca8a282f94392bb2746ed7b8b8ba7 deleted and replaced by tablet d8766c4d56fa4689a69b153ea7a613ee
I20261118 02:55:20.901203 4982 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Sending DeleteTablet for 3 replicas of tablet 8ae679d2feb04b95ab2f4fb51e443d64
I20261118 02:55:20.901605 4982 catalog_manager.cc:6936] ReplaceTablet: tablet 8ae679d2feb04b95ab2f4fb51e443d64 deleted and replaced by tablet c46a0eff7633462fb3719143c1f7cfbb
I20261118 02:55:20.904330 4983 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37916:
table { table_name: "default.Table_8ae376d4d4d64414b5c08dc3c2e58bc4" } new_table_name: "default.Table_5091d82298ca44bea5335d4989eb02d8" modify_external_catalogs: true
I20261118 02:55:20.904771 4984 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37916:
table { table_name: "default.Table_a14cb6414555446286bf35bc5c6fb910" } new_table_name: "default.Table_324a2600fef947748ef03e27f7cb7e09" modify_external_catalogs: true
I20261118 02:55:20.905241 4982 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37916:
table { table_name: "default.Table_2e3917ff98f2474984580e61caa9c17b" } new_table_name: "default.Table_3c225da1f7fa435083909e053ec9751a" modify_external_catalogs: true
@ 0x7fcb80459d49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
@ 0x7fcb6c41cdc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcb6faf76db start_thread at ??:0
@ 0x7fcb6856761f clone at ??:0
I20261118 02:55:20.918442 3595 tablet_service.cc:1547] Processing DeleteTablet for tablet c0713dcf79f54873a99d7556ad8f1454 with delete_type TABLET_DATA_DELETED (replaced by tablet fbdc4c7b574e4f94b816fb0c0b14c185) from {username='slave'} at 127.0.0.1:46596
I20261118 02:55:20.918504 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet 973831392ec24e0681c3d4fffd8bb199 with delete_type TABLET_DATA_DELETED (replaced by tablet 462a9be0ae114b01a9c69416185803f8) from {username='slave'} at 127.0.0.1:46596
I20261118 02:55:20.918907 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet 1d347c0ae53241708ec623562f4740d4 with delete_type TABLET_DATA_DELETED (replaced by tablet a67bc60c7ecb4b7e8e03e04267bb91a5) from {username='slave'} at 127.0.0.1:46596
I20261118 02:55:20.919237 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet ffe8d4c111d14054b76f9c7724072a94 with delete_type TABLET_DATA_DELETED (replaced by tablet b119ead4bb6548b4a37da02ed6daa6a8) from {username='slave'} at 127.0.0.1:46596
I20261118 02:55:20.919586 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet b82ca8a282f94392bb2746ed7b8b8ba7 with delete_type TABLET_DATA_DELETED (replaced by tablet d8766c4d56fa4689a69b153ea7a613ee) from {username='slave'} at 127.0.0.1:46596
I20261118 02:55:20.919940 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet 8ae679d2feb04b95ab2f4fb51e443d64 with delete_type TABLET_DATA_DELETED (replaced by tablet c46a0eff7633462fb3719143c1f7cfbb) from {username='slave'} at 127.0.0.1:46596
I20261118 02:55:20.920485 5271 tablet_replica.cc:333] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:20.920682 5271 raft_consensus.cc:2261] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:20.921073 5271 raft_consensus.cc:2290] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:20.921703 5271 ts_tablet_manager.cc:1962] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:20.930291 5271 ts_tablet_manager.cc:1975] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:20.931552 5271 log.cc:1199] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/c0713dcf79f54873a99d7556ad8f1454
I20261118 02:55:20.931989 5271 ts_tablet_manager.cc:1996] T c0713dcf79f54873a99d7556ad8f1454 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:20.933106 5271 tablet_replica.cc:333] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:20.933429 5271 raft_consensus.cc:2261] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:20.933599 5271 raft_consensus.cc:2290] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:20.934137 5271 ts_tablet_manager.cc:1962] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:20.940752 5271 ts_tablet_manager.cc:1975] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:20.940886 5271 log.cc:1199] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/973831392ec24e0681c3d4fffd8bb199
I20261118 02:55:20.941252 5271 ts_tablet_manager.cc:1996] T 973831392ec24e0681c3d4fffd8bb199 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:20.942377 5271 tablet_replica.cc:333] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:20.942574 5271 raft_consensus.cc:2261] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:20.942960 5271 raft_consensus.cc:2290] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:20.943509 5271 ts_tablet_manager.cc:1962] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:20.950045 5271 ts_tablet_manager.cc:1975] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:20.950150 5271 log.cc:1199] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/1d347c0ae53241708ec623562f4740d4
I20261118 02:55:20.950510 5271 ts_tablet_manager.cc:1996] T 1d347c0ae53241708ec623562f4740d4 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:20.951570 5271 tablet_replica.cc:333] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:20.951766 5271 raft_consensus.cc:2261] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:20.951932 5271 raft_consensus.cc:2290] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:20.952513 5271 ts_tablet_manager.cc:1962] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:20.958586 5271 ts_tablet_manager.cc:1975] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:20.958710 5271 log.cc:1199] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/ffe8d4c111d14054b76f9c7724072a94
I20261118 02:55:20.959069 5271 ts_tablet_manager.cc:1996] T ffe8d4c111d14054b76f9c7724072a94 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:20.960246 5271 tablet_replica.cc:333] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:20.960425 5271 raft_consensus.cc:2261] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:20.960587 5271 raft_consensus.cc:2290] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:20.961115 5271 ts_tablet_manager.cc:1962] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:20.964831 3800 heartbeater.cc:345] Connected to a master server at 127.2.229.126:38025
I20261118 02:55:20.967343 5183 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" instance_seqno: 1794970499028671) as {username='slave'} at 127.2.229.66:33609; Asking this server to re-register.
I20261118 02:55:20.967505 5271 ts_tablet_manager.cc:1975] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:20.967613 5271 log.cc:1199] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/b82ca8a282f94392bb2746ed7b8b8ba7
I20261118 02:55:20.968010 5271 ts_tablet_manager.cc:1996] T b82ca8a282f94392bb2746ed7b8b8ba7 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:20.968227 3800 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:20.969126 5271 tablet_replica.cc:333] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:20.969329 5271 raft_consensus.cc:2261] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:20.969493 5271 raft_consensus.cc:2290] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:20.970058 5271 ts_tablet_manager.cc:1962] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:20.971170 5183 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:20.977316 5271 ts_tablet_manager.cc:1975] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:20.977496 5271 log.cc:1199] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/8ae679d2feb04b95ab2f4fb51e443d64
I20261118 02:55:20.977895 5271 ts_tablet_manager.cc:1996] T 8ae679d2feb04b95ab2f4fb51e443d64 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
W20261118 02:55:21.110530 3956 connection.cc:573] client connection to 127.2.229.124:36695 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:21.110631 3552 connection.cc:573] server connection from 127.0.0.1:46596 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:21.111089 3956 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.229.124:36695)
W20261118 02:55:21.111009 5267 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:20.871330 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.2.229.67:42237)
1118 02:55:20.904401 (+ 33071us) server_negotiation.cc:207] Beginning negotiation
1118 02:55:20.904408 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1118 02:55:20.904427 (+ 19us) server_negotiation.cc:408] Connection header received
1118 02:55:20.904490 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1118 02:55:20.904495 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1118 02:55:20.904569 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1118 02:55:20.906533 (+ 1964us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1118 02:55:20.906784 (+ 251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1118 02:55:20.911023 (+ 4239us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1118 02:55:21.110959 (+199936us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36202: BlockingRecv error: recv got EOF from 127.0.0.1:36202 (error 108)
Metrics: {"server-negotiator.queue_time_us":32949,"thread_start_us":43980,"threads_started":1}
W20261118 02:55:21.111656 3956 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.229.124:36695)
W20261118 02:55:21.111814 3956 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.229.124:36695)
W20261118 02:55:21.111968 3956 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.229.124:36695)
W20261118 02:55:21.110054 3690 connection.cc:573] client connection to 127.2.229.124:36695 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:21.115104 5266 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:20.870711 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.2.229.66:40377)
1118 02:55:20.916921 (+ 46210us) server_negotiation.cc:207] Beginning negotiation
1118 02:55:20.916927 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1118 02:55:20.916949 (+ 22us) server_negotiation.cc:408] Connection header received
1118 02:55:20.917018 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1118 02:55:20.917024 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1118 02:55:20.917122 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1118 02:55:20.917229 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1118 02:55:21.115058 (+197829us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41084: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":46083,"thread_start_us":45991,"threads_started":1}
W20261118 02:55:21.117075 4790 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:21.116414 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.2.229.124:36695 (local address 127.0.0.1:37946)
1118 02:55:21.116574 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1118 02:55:21.116592 (+ 18us) client_negotiation.cc:175] Beginning negotiation
1118 02:55:21.116705 (+ 113us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 02:55:21.117013 (+ 308us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
W20261118 02:55:21.160531 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:21.399070 3664 heartbeater.cc:500] Master 127.2.229.124:36695 was elected leader, sending a full tablet report...
I20261118 02:55:21.411652 5282 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:21.411839 5282 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:21.413774 5282 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:21.414546 5166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:21.415915 5166 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:21.416153 5167 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:21.416280 5167 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d; no voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:21.416625 5282 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20261118 02:55:21.528079 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 1: this message will repeat every 5th retry.
W20261118 02:55:21.747941 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 1: this message will repeat every 5th retry.
W20261118 02:55:22.006570 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:22.064800 5290 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:22.064986 5290 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:22.065735 5290 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:22.067557 5166 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:22.067719 5167 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:22.067824 5167 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d; no voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:22.068064 5290 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20261118 02:55:22.742434 5299 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:22.742620 5299 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:22.743258 5299 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:22.745391 5166 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:22.745563 5167 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:22.745682 5167 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d; no voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:22.745937 5299 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20261118 02:55:22.935164 5296 raft_consensus.cc:499] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1fdf7d7ece324fefb3a0c376add5b7e8)
I20261118 02:55:22.935381 5296 raft_consensus.cc:521] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:22.936177 5296 leader_election.cc:290] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:22.936852 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c0713dcf79f54873a99d7556ad8f1454" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
W20261118 02:55:22.937325 3691 leader_election.cc:343] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454
I20261118 02:55:22.937566 3690 leader_election.cc:304] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f
I20261118 02:55:22.937970 5296 raft_consensus.cc:2771] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:23.373956 5223 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:55:23.374172 5223 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:23.375036 5223 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:23.375814 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:55:23.376040 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:23.376251 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:23.376608 5223 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:23.552467 5305 raft_consensus.cc:499] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1fdf7d7ece324fefb3a0c376add5b7e8)
I20261118 02:55:23.552685 5305 raft_consensus.cc:521] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:23.553629 5305 leader_election.cc:290] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:23.554548 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c0713dcf79f54873a99d7556ad8f1454" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
W20261118 02:55:23.554819 3838 leader_election.cc:343] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454
I20261118 02:55:23.554966 3751 raft_consensus.cc:2490] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:23.555377 3836 leader_election.cc:304] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:23.555620 5305 raft_consensus.cc:2826] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261118 02:55:23.555733 5305 raft_consensus.cc:499] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1fdf7d7ece324fefb3a0c376add5b7e8)
I20261118 02:55:23.555819 5305 raft_consensus.cc:3096] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Advancing to term 2
I20261118 02:55:23.558722 5305 raft_consensus.cc:521] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:23.559363 5305 leader_election.cc:290] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 2 election: Requested vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:23.559871 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c0713dcf79f54873a99d7556ad8f1454" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:23.560138 3751 raft_consensus.cc:3096] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Advancing to term 2
W20261118 02:55:23.560264 3838 leader_election.cc:343] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454
I20261118 02:55:23.563066 3751 raft_consensus.cc:2490] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 2.
I20261118 02:55:23.563484 3836 leader_election.cc:304] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8
I20261118 02:55:23.563917 5305 raft_consensus.cc:2826] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 2 FOLLOWER]: Leader election won for term 2
I20261118 02:55:23.564332 5305 raft_consensus.cc:705] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [term 2 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:23.564640 5305 consensus_queue.cc:238] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:23.566792 3751 raft_consensus.cc:1285] T c0713dcf79f54873a99d7556ad8f1454 P 99a39b1d9d814deeb3a8a644e9b392ef [term 2 FOLLOWER]: Refusing update from remote peer 3f1973986ec1466eab5bec01dd81654f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20261118 02:55:23.567268 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:23.567396 5305 consensus_queue.cc:1049] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261118 02:55:23.589186 5223 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:55:23.589414 5223 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:23.590181 5223 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:23.590929 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:55:23.591190 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:55:23.591316 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:23.591562 5223 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:23.960971 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.125:38339
--webserver_interface=127.2.229.125
--webserver_port=39093
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 02:55:23.990309 5316 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:23.990501 5316 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:23.991230 5316 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:23.993335 5166 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:23.993484 5167 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:23.993597 5167 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d; no voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:23.993933 5316 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20261118 02:55:24.303120 5315 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:55:24.303491 5315 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:55:24.303613 5315 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:55:24.313185 5315 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:55:24.313288 5315 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:55:24.313366 5315 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:55:24.313433 5315 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:55:24.323764 5315 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.125:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.229.125
--webserver_port=39093
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:55:24.325577 5315 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:55:24.327409 5315 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:55:24.335559 5331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:24.335638 5330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:24.336400 5333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:55:24.336812 5315 server_base.cc:1061] running on GCE node
I20261118 02:55:24.337532 5315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:55:24.339097 5315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:55:24.340365 5315 hybrid_clock.cc:648] HybridClock initialized: now 1794970524340321 us; error 41 us; skew 500 ppm
I20261118 02:55:24.340730 5315 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:55:24.343422 5315 webserver.cc:577] Webserver started at http://127.2.229.125:39093/ using document root <none> and password file <none>
I20261118 02:55:24.344174 5315 fs_manager.cc:362] Metadata directory not provided
I20261118 02:55:24.344302 5315 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:55:24.350091 5315 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20261118 02:55:24.353492 5339 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:55:24.355297 5315 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20261118 02:55:24.355468 5315 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
uuid: "457eb38cba564ecd9f6f11245172260b"
format_stamp: "Formatted at 2026-11-18 02:54:56 on dist-test-slave-49p1"
I20261118 02:55:24.356257 5315 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:55:24.380457 5315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:55:24.381372 5315 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:55:24.381637 5315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:55:24.399927 5315 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.125:38339
I20261118 02:55:24.399945 5390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.125:38339 every 8 connection(s)
I20261118 02:55:24.401556 5315 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-1/data/info.pb
I20261118 02:55:24.408874 5391 sys_catalog.cc:263] Verifying existing consensus state
I20261118 02:55:24.409404 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 5315
I20261118 02:55:24.413405 5391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap starting.
I20261118 02:55:24.435981 5391 log.cc:826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:24.511309 5391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap replayed 1/1 log segments. Stats: ops{read=106 overwritten=0 applied=106 ignored=0} inserts{seen=70 ignored=0} mutations{seen=93 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:24.511799 5391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Bootstrap complete.
I20261118 02:55:24.517663 5391 raft_consensus.cc:360] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:24.518399 5391 raft_consensus.cc:748] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457eb38cba564ecd9f6f11245172260b, State: Initialized, Role: FOLLOWER
I20261118 02:55:24.519016 5391 consensus_queue.cc:261] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 106, Last appended: 7.106, Last appended by leader: 106, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:24.519794 5396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:24.520058 5396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:24.520825 5391 sys_catalog.cc:565] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:55:24.568730 5410 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:24.568949 5410 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: loading cluster ID for follower catalog manager: success
I20261118 02:55:24.574469 5410 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: acquiring CA information for follower catalog manager: success
I20261118 02:55:24.581694 5410 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:55:24.576993 5315 master_runner.cc:187] Error getting master registration for 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111),
I20261118 02:55:24.582908 5315 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:24.651752 3949 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:24.656266 5356 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" instance_seqno: 1794970499527172) as {username='slave'} at 127.2.229.67:38051; Asking this server to re-register.
I20261118 02:55:24.657300 3949 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:24.659678 5356 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:24.765259 3802 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:24.768347 5356 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" instance_seqno: 1794970499028671) as {username='slave'} at 127.2.229.66:45571; Asking this server to re-register.
I20261118 02:55:24.769181 3802 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:24.771147 5356 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:24.871848 3665 heartbeater.cc:345] Connected to a master server at 127.2.229.125:38339
I20261118 02:55:24.873872 5356 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" instance_seqno: 1794970498532688) as {username='slave'} at 127.2.229.65:56813; Asking this server to re-register.
I20261118 02:55:24.874491 3665 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:24.875998 5356 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:25.036291 5421 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:25.036616 5421 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:25.038511 5421 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:25.039032 5193 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 8 candidate_status { last_received { term: 7 index: 106 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
W20261118 02:55:25.039248 5342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:25.039367 5193 raft_consensus.cc:2415] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 457eb38cba564ecd9f6f11245172260b in current term 8: Already voted for candidate 813a6937cea442ef81645fccc32207f4 in this term.
W20261118 02:55:25.040669 5342 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:25.040836 5342 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:25.041169 5421 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 7 FOLLOWER]: Advancing to term 8
I20261118 02:55:25.045043 5421 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 02:55:25.565763 5426 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:25.565958 5426 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:25.566581 5426 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:25.567063 5193 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 9 candidate_status { last_received { term: 7 index: 106 } } ignore_live_leader: false dest_uuid: "26c6b20fa203404a88463ea27bdd3c7d" is_pre_election: true
I20261118 02:55:25.567327 5193 raft_consensus.cc:2432] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 457eb38cba564ecd9f6f11245172260b for term 9 because replica has last-logged OpId of term: 8 index: 113, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 106.
W20261118 02:55:25.568538 5342 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:25.568653 5342 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:25.568904 5426 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20261118 02:55:25.699878 5146 master_runner.cc:187] Error getting master registration for 127.2.229.124:36695: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111),
I20261118 02:55:25.700084 5146 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:26.314917 5436 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
I20261118 02:55:26.315114 5436 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:26.315905 5436 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:26.316627 5366 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 9 candidate_status { last_received { term: 8 index: 113 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b" is_pre_election: true
I20261118 02:55:26.316994 5366 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 8.
I20261118 02:55:26.317548 5166 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b; no voters:
I20261118 02:55:26.317836 5436 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20261118 02:55:26.317984 5436 raft_consensus.cc:499] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 813a6937cea442ef81645fccc32207f4)
W20261118 02:55:26.318037 5167 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:26.318125 5436 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 8 FOLLOWER]: Advancing to term 9
I20261118 02:55:26.320196 5436 raft_consensus.cc:521] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:26.320786 5436 leader_election.cc:290] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 election: Requested vote from peers 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:26.321272 5366 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c6b20fa203404a88463ea27bdd3c7d" candidate_term: 9 candidate_status { last_received { term: 8 index: 113 } } ignore_live_leader: false dest_uuid: "457eb38cba564ecd9f6f11245172260b"
I20261118 02:55:26.321465 5366 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 8 FOLLOWER]: Advancing to term 9
W20261118 02:55:26.322074 5167 leader_election.cc:336] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:26.323616 5366 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c6b20fa203404a88463ea27bdd3c7d in term 9.
I20261118 02:55:26.324146 5166 leader_election.cc:304] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b; no voters: 813a6937cea442ef81645fccc32207f4
I20261118 02:55:26.324491 5436 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 9 FOLLOWER]: Leader election won for term 9
I20261118 02:55:26.324888 5436 raft_consensus.cc:705] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [term 9 LEADER]: Becoming Leader. State: Replica: 26c6b20fa203404a88463ea27bdd3c7d, State: Running, Role: LEADER
I20261118 02:55:26.325345 5436 consensus_queue.cc:238] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 8.113, Last appended by leader: 113, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:26.326917 5439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 9 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:26.327276 5439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:26.328112 5443 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 02:55:26.329102 5366 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Refusing update from remote peer 26c6b20fa203404a88463ea27bdd3c7d: Log matching property violated. Preceding OpId in replica: term: 7 index: 106. Preceding OpId from leader: term: 9 index: 114. (index mismatch)
W20261118 02:55:26.329769 5167 consensus_peers.cc:597] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d -> Peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Couldn't send request to peer 813a6937cea442ef81645fccc32207f4. Status: Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:26.330085 5441 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 106, Time since last communication: 0.000s
I20261118 02:55:26.331959 5443 catalog_manager.cc:694] Loaded metadata for table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190]
I20261118 02:55:26.332523 5443 catalog_manager.cc:694] Loaded metadata for table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c]
I20261118 02:55:26.332958 5443 catalog_manager.cc:694] Loaded metadata for table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]
I20261118 02:55:26.333416 5443 catalog_manager.cc:694] Loaded metadata for table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]
I20261118 02:55:26.333820 5443 catalog_manager.cc:694] Loaded metadata for table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]
I20261118 02:55:26.334268 5443 catalog_manager.cc:694] Loaded metadata for table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]
I20261118 02:55:26.334888 5443 catalog_manager.cc:694] Loaded metadata for table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da]
I20261118 02:55:26.335407 5443 catalog_manager.cc:694] Loaded metadata for table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]
I20261118 02:55:26.335824 5443 catalog_manager.cc:694] Loaded metadata for table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d]
I20261118 02:55:26.336251 5443 catalog_manager.cc:694] Loaded metadata for table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]
I20261118 02:55:26.336658 5443 catalog_manager.cc:694] Loaded metadata for table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]
I20261118 02:55:26.337522 5443 catalog_manager.cc:694] Loaded metadata for table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b]
I20261118 02:55:26.337939 5443 catalog_manager.cc:694] Loaded metadata for table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]
I20261118 02:55:26.338344 5443 catalog_manager.cc:694] Loaded metadata for table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]
I20261118 02:55:26.338749 5443 catalog_manager.cc:694] Loaded metadata for table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]
I20261118 02:55:26.339143 5443 catalog_manager.cc:694] Loaded metadata for table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]
I20261118 02:55:26.339550 5443 catalog_manager.cc:694] Loaded metadata for table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]
I20261118 02:55:26.339964 5443 catalog_manager.cc:694] Loaded metadata for table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1]
I20261118 02:55:26.340418 5443 catalog_manager.cc:694] Loaded metadata for table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f]
I20261118 02:55:26.340855 5443 catalog_manager.cc:694] Loaded metadata for table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]
I20261118 02:55:26.344050 5443 tablet_loader.cc:96] loaded metadata for tablet 01778879aaa44ea9855fae67cfb3952d (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7])
I20261118 02:55:26.344540 5443 tablet_loader.cc:96] loaded metadata for tablet 0e901a03b90e4c978eb98be87163cfe5 (table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190])
I20261118 02:55:26.344913 5443 tablet_loader.cc:96] loaded metadata for tablet 1904140da78b4567a9decb250c6f74a3 (table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f])
I20261118 02:55:26.345512 5443 tablet_loader.cc:96] loaded metadata for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db])
I20261118 02:55:26.345849 5443 tablet_loader.cc:96] loaded metadata for tablet 2c92955aa65142f1bd20f4b8f3321500 (table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d])
I20261118 02:55:26.346398 5443 tablet_loader.cc:96] loaded metadata for tablet 462a9be0ae114b01a9c69416185803f8 (table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139])
I20261118 02:55:26.346750 5443 tablet_loader.cc:96] loaded metadata for tablet 46f05790dbdc4b8185a7b0362f604cf8 (table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf])
I20261118 02:55:26.348302 5443 tablet_loader.cc:96] loaded metadata for tablet 65bd6fcaed5b47c3a2478d375ace9636 (table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73])
I20261118 02:55:26.348794 5445 mvcc.cc:204] Tried to move back new op lower bound from 7352199253305876480 to 7352199253043658752. Current Snapshot: MvccSnapshot[applied={T|T < 7352199253305876480}]
I20261118 02:55:26.348979 5443 tablet_loader.cc:96] loaded metadata for tablet 72f427c72a554ac1a0a2f2c3a78865da (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b])
I20261118 02:55:26.349395 5443 tablet_loader.cc:96] loaded metadata for tablet 78189cccd457466bb5d01231038fec24 (table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da])
I20261118 02:55:26.349987 5443 tablet_loader.cc:96] loaded metadata for tablet 8bed657d0da64d95851f8a86216b6c2c (table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c])
I20261118 02:55:26.351516 5443 tablet_loader.cc:96] loaded metadata for tablet a67bc60c7ecb4b7e8e03e04267bb91a5 (table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b])
I20261118 02:55:26.352082 5443 tablet_loader.cc:96] loaded metadata for tablet ab61b43e639a4e75bc2e7009ea86a64f (table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67])
I20261118 02:55:26.352458 5443 tablet_loader.cc:96] loaded metadata for tablet b119ead4bb6548b4a37da02ed6daa6a8 (table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66])
I20261118 02:55:26.352864 5443 tablet_loader.cc:96] loaded metadata for tablet b129518ce23c45758ebcaa55410757d8 (table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1])
I20261118 02:55:26.353163 5444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6b20fa203404a88463ea27bdd3c7d. Latest consensus state: current_term: 9 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:26.353422 5444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:26.353843 5443 tablet_loader.cc:96] loaded metadata for tablet c46a0eff7633462fb3719143c1f7cfbb (table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d])
I20261118 02:55:26.354539 5443 tablet_loader.cc:96] loaded metadata for tablet ccf78258c9634540a178855a142f4d4b (table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e])
I20261118 02:55:26.354481 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:26.354820 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:26.355557 5443 tablet_loader.cc:96] loaded metadata for tablet d8766c4d56fa4689a69b153ea7a613ee (table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d])
I20261118 02:55:26.355520 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:26.355824 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:26.356273 5443 tablet_loader.cc:96] loaded metadata for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6])
I20261118 02:55:26.356190 5442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "26c6b20fa203404a88463ea27bdd3c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:26.356467 5442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:26.357071 5443 tablet_loader.cc:96] loaded metadata for tablet fbdc4c7b574e4f94b816fb0c0b14c185 (table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec])
I20261118 02:55:26.357542 5443 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 02:55:26.358835 5443 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:26.358901 5443 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:55:26.360419 5443 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:55:26.361577 5443 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Loaded TSK: 0
I20261118 02:55:26.362411 5443 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 02:55:26.363606 5443 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:55:26.364877 5443 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: 20
I20261118 02:55:26.377704 5182 master_service.cc:965] ReplaceTablet: received request to replace tablet 01778879aaa44ea9855fae67cfb3952d from {username='slave'} at 127.0.0.1:59696
I20261118 02:55:26.377913 5183 master_service.cc:965] ReplaceTablet: received request to replace tablet 65bd6fcaed5b47c3a2478d375ace9636 from {username='slave'} at 127.0.0.1:59696
I20261118 02:55:26.378772 5177 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59696:
table { table_name: "default.Table_a7f3710d39fb447da94e16d4ab3933fd" } modify_external_catalogs: true
I20261118 02:55:26.379175 5179 master_service.cc:965] ReplaceTablet: received request to replace tablet 8ae679d2feb04b95ab2f4fb51e443d64 from {username='slave'} at 127.0.0.1:59696
I20261118 02:55:26.379484 5181 master_service.cc:965] ReplaceTablet: received request to replace tablet ffe8d4c111d14054b76f9c7724072a94 from {username='slave'} at 127.0.0.1:59696
I20261118 02:55:26.380021 5179 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59696:
table { table_name: "default.Table_571d1360811c42dc815efb439b2b52e6" } modify_external_catalogs: true
I20261118 02:55:26.380180 5180 master_service.cc:965] ReplaceTablet: received request to replace tablet 973831392ec24e0681c3d4fffd8bb199 from {username='slave'} at 127.0.0.1:59696
I20261118 02:55:26.380275 5179 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59696:
table { table_name: "default.Table_571d1360811c42dc815efb439b2b52e6" } modify_external_catalogs: true
I20261118 02:55:26.380223 5174 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59696:
table { table_name: "default.Table_28c62f411da74545aa42dad7325bc064" } modify_external_catalogs: true
I20261118 02:55:26.380674 5174 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59696:
table { table_name: "default.Table_28c62f411da74545aa42dad7325bc064" } modify_external_catalogs: true
I20261118 02:55:26.380440 5176 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59696:
table { table_name: "default.Table_4f7796d5d5d54def843b971e66d15ea6" } new_table_name: "default.Table_79eb11797e3f4bc2bffe161096300f7a" modify_external_catalogs: true
I20261118 02:55:26.378835 5178 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59696:
name: "default.Table_fd4fcac1f1d946fa898176f1d5e87e9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:26.380759 5177 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59696:
table { table_name: "default.Table_a7f3710d39fb447da94e16d4ab3933fd" } modify_external_catalogs: true
I20261118 02:55:26.379794 5175 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59696:
name: "default.Table_e39f59e2fdbb47fda4c10beba29ad8f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:26.383098 5180 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59696:
name: "default.Table_52fca7a2e65f4e679bbef9de3df569f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:26.383123 5181 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59696:
name: "default.Table_40868189cecf45d0b705461c4328ee17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20261118 02:55:26.390275 5238 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 21, previous 21; dumping first 2 out of 15 received events: NotificationEvent(eventId=21, eventTime=1794970503, eventType=CREATE_TABLE, dbName=default, tableName=table_f0ae490a0df64e1097e144c7a370df61, message={"server":"","servicePrincipal":"","db":"default","table":"table_f0ae490a0df64e1097e144c7a370df61","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f0ae490a0df64e1097e144c7a370df61\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970503},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_f0ae490a0df64e1097e144c7a370df61\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f0ae490a0df64e1097e144c7a370df61\",\"kudu.table_id\":\"a59b9425c01049269883cf7767936c67\",\"transient_lastDdlTime\":\"1794970503\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1794970503, eventType=DROP_TABLE, dbName=default, tableName=table_c113c2b57e17479e92e241234a241af7, message={"server":"","servicePrincipal":"","db":"default","table":"table_c113c2b57e17479e92e241234a241af7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c113c2b57e17479e92e241234a241af7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970500},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_c113c2b57e17479e92e241234a241af7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c113c2b57e17479e92e241234a241af7\",\"kudu.table_id\":\"da0cb25ccfed426ca61f0dd9f93d30ec\",\"transient_lastDdlTime\":\"1794970500\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f55745a2d5d google::LogMessage::Fail() at ??:0
@ 0x7f55745a6b23 google::LogMessage::SendToLog() at ??:0
I20261118 02:55:26.395017 5182 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet 01778879aaa44ea9855fae67cfb3952d
@ 0x7f55745a275c google::LogMessage::Flush() at ??:0
I20261118 02:55:26.397074 5182 catalog_manager.cc:6936] ReplaceTablet: tablet 01778879aaa44ea9855fae67cfb3952d deleted and replaced by tablet b400925bbe6f4639b3d9e76cbb1ca681
I20261118 02:55:26.399232 5183 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 26c6b20fa203404a88463ea27bdd3c7d: Sending DeleteTablet for 3 replicas of tablet 65bd6fcaed5b47c3a2478d375ace9636
I20261118 02:55:26.399708 5183 catalog_manager.cc:6936] ReplaceTablet: tablet 65bd6fcaed5b47c3a2478d375ace9636 deleted and replaced by tablet c1f0147a1efb4f4f813478f119043bec
@ 0x7f55745a29a1 google::LogMessage::~LogMessage() at ??:0
I20261118 02:55:26.422303 5182 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59696:
name: "default.Table_ee3815d9524243ec886bd171ccaff9e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:26.422988 5183 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59696:
name: "default.Table_bdbed231860946c2bf9b514f724bc3e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f5589a229a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 02:55:26.440920 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet 01778879aaa44ea9855fae67cfb3952d with delete_type TABLET_DATA_DELETED (replaced by tablet b400925bbe6f4639b3d9e76cbb1ca681) from {username='slave'} at 127.0.0.1:35078
I20261118 02:55:26.441577 5461 tablet_replica.cc:333] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:26.441788 5461 raft_consensus.cc:2261] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:26.441967 3732 tablet_service.cc:1547] Processing DeleteTablet for tablet 01778879aaa44ea9855fae67cfb3952d with delete_type TABLET_DATA_DELETED (replaced by tablet b400925bbe6f4639b3d9e76cbb1ca681) from {username='slave'} at 127.0.0.1:50108
I20261118 02:55:26.442023 3881 tablet_service.cc:1547] Processing DeleteTablet for tablet 65bd6fcaed5b47c3a2478d375ace9636 with delete_type TABLET_DATA_DELETED (replaced by tablet c1f0147a1efb4f4f813478f119043bec) from {username='slave'} at 127.0.0.1:35078
I20261118 02:55:26.442066 5461 raft_consensus.cc:2290] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:26.442253 3733 tablet_service.cc:1547] Processing DeleteTablet for tablet 65bd6fcaed5b47c3a2478d375ace9636 with delete_type TABLET_DATA_DELETED (replaced by tablet c1f0147a1efb4f4f813478f119043bec) from {username='slave'} at 127.0.0.1:50108
I20261118 02:55:26.442514 3734 tablet_service.cc:1500] Processing CreateTablet for tablet c46a0eff7633462fb3719143c1f7cfbb (DEFAULT_TABLE table=default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.442624 3735 tablet_service.cc:1500] Processing CreateTablet for tablet c1f0147a1efb4f4f813478f119043bec (DEFAULT_TABLE table=default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.442960 3734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46a0eff7633462fb3719143c1f7cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.443212 3733 tablet_service.cc:1500] Processing CreateTablet for tablet b119ead4bb6548b4a37da02ed6daa6a8 (DEFAULT_TABLE table=default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.443612 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b119ead4bb6548b4a37da02ed6daa6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.443516 3731 tablet_service.cc:1500] Processing CreateTablet for tablet d8766c4d56fa4689a69b153ea7a613ee (DEFAULT_TABLE table=default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.444670 3731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8766c4d56fa4689a69b153ea7a613ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.445508 3735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f0147a1efb4f4f813478f119043bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.449819 5461 ts_tablet_manager.cc:1962] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:26.449874 3881 tablet_service.cc:1500] Processing CreateTablet for tablet c46a0eff7633462fb3719143c1f7cfbb (DEFAULT_TABLE table=default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.450317 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46a0eff7633462fb3719143c1f7cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.452347 5462 tablet_replica.cc:333] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:26.453279 3877 tablet_service.cc:1500] Processing CreateTablet for tablet c1f0147a1efb4f4f813478f119043bec (DEFAULT_TABLE table=default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.453666 3877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f0147a1efb4f4f813478f119043bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.453908 3882 tablet_service.cc:1500] Processing CreateTablet for tablet b400925bbe6f4639b3d9e76cbb1ca681 (DEFAULT_TABLE table=default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.453910 3878 tablet_service.cc:1500] Processing CreateTablet for tablet d8766c4d56fa4689a69b153ea7a613ee (DEFAULT_TABLE table=default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.454357 3882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b400925bbe6f4639b3d9e76cbb1ca681. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.453676 3880 tablet_service.cc:1500] Processing CreateTablet for tablet b119ead4bb6548b4a37da02ed6daa6a8 (DEFAULT_TABLE table=default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.459160 3878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8766c4d56fa4689a69b153ea7a613ee. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f5589a21251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20261118 02:55:26.465845 3595 tablet_service.cc:1547] Processing DeleteTablet for tablet 01778879aaa44ea9855fae67cfb3952d with delete_type TABLET_DATA_DELETED (replaced by tablet b400925bbe6f4639b3d9e76cbb1ca681) from {username='slave'} at 127.0.0.1:46360
I20261118 02:55:26.465983 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet 65bd6fcaed5b47c3a2478d375ace9636 with delete_type TABLET_DATA_DELETED (replaced by tablet c1f0147a1efb4f4f813478f119043bec) from {username='slave'} at 127.0.0.1:46360
I20261118 02:55:26.466496 3597 tablet_service.cc:1500] Processing CreateTablet for tablet b119ead4bb6548b4a37da02ed6daa6a8 (DEFAULT_TABLE table=default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.466737 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b119ead4bb6548b4a37da02ed6daa6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.466688 3594 tablet_service.cc:1500] Processing CreateTablet for tablet c46a0eff7633462fb3719143c1f7cfbb (DEFAULT_TABLE table=default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.466881 3597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b119ead4bb6548b4a37da02ed6daa6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.467092 3594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46a0eff7633462fb3719143c1f7cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.472448 3598 tablet_service.cc:1500] Processing CreateTablet for tablet c1f0147a1efb4f4f813478f119043bec (DEFAULT_TABLE table=default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.472841 3598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f0147a1efb4f4f813478f119043bec. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f5589a28ecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20261118 02:55:26.476421 3596 tablet_service.cc:1500] Processing CreateTablet for tablet d8766c4d56fa4689a69b153ea7a613ee (DEFAULT_TABLE table=default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.476763 3596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8766c4d56fa4689a69b153ea7a613ee. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f5589a28d49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
I20261118 02:55:26.480116 3592 tablet_service.cc:1500] Processing CreateTablet for tablet b400925bbe6f4639b3d9e76cbb1ca681 (DEFAULT_TABLE table=default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.480607 3592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b400925bbe6f4639b3d9e76cbb1ca681. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.453014 3730 tablet_service.cc:1500] Processing CreateTablet for tablet b400925bbe6f4639b3d9e76cbb1ca681 (DEFAULT_TABLE table=default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 02:55:26.481835 3730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b400925bbe6f4639b3d9e76cbb1ca681. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 02:55:26.483476 5462 raft_consensus.cc:2261] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
@ 0x7f55759ebdc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f55790c66db start_thread at ??:0
@ 0x7f5571b3661f clone at ??:0
I20261118 02:55:26.497404 5462 raft_consensus.cc:2290] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:26.498037 5462 ts_tablet_manager.cc:1962] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
W20261118 02:55:26.502343 3838 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.229.65:33705, user_credentials={real_user=slave}} blocked reactor thread for 47933.7us
I20261118 02:55:26.504607 5470 tablet_bootstrap.cc:492] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:26.507344 5469 tablet_bootstrap.cc:492] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:26.510207 5469 tablet_bootstrap.cc:654] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.511461 5467 tablet_replica.cc:333] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:26.511698 5467 raft_consensus.cc:2261] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:26.512156 5467 raft_consensus.cc:2290] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:26.512800 5467 ts_tablet_manager.cc:1962] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:26.513556 5461 ts_tablet_manager.cc:1975] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:26.513674 5461 log.cc:1199] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/01778879aaa44ea9855fae67cfb3952d
I20261118 02:55:26.514061 5461 ts_tablet_manager.cc:1996] T 01778879aaa44ea9855fae67cfb3952d P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:26.515072 5461 tablet_replica.cc:333] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: stopping tablet replica
I20261118 02:55:26.515334 5461 raft_consensus.cc:2261] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Raft consensus shutting down.
I20261118 02:55:26.515669 5461 raft_consensus.cc:2290] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:26.516408 5470 tablet_bootstrap.cc:654] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.517941 5468 tablet_bootstrap.cc:492] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:26.519225 5462 ts_tablet_manager.cc:1975] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:26.519313 5462 log.cc:1199] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/01778879aaa44ea9855fae67cfb3952d
I20261118 02:55:26.519587 5462 ts_tablet_manager.cc:1996] T 01778879aaa44ea9855fae67cfb3952d P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:26.519951 5468 tablet_bootstrap.cc:654] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.520807 5462 tablet_replica.cc:333] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: stopping tablet replica
I20261118 02:55:26.521095 5462 raft_consensus.cc:2261] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:26.521545 5462 raft_consensus.cc:2290] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:26.521744 5461 ts_tablet_manager.cc:1962] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:26.522405 5470 tablet_bootstrap.cc:492] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:26.522540 5470 ts_tablet_manager.cc:1449] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20261118 02:55:26.523609 5470 raft_consensus.cc:360] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.523804 5470 raft_consensus.cc:386] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.523883 5470 raft_consensus.cc:748] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.527484 5462 ts_tablet_manager.cc:1962] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:26.528033 5467 ts_tablet_manager.cc:1975] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20261118 02:55:26.528185 5467 log.cc:1199] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/01778879aaa44ea9855fae67cfb3952d
I20261118 02:55:26.528590 5467 ts_tablet_manager.cc:1996] T 01778879aaa44ea9855fae67cfb3952d P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:26.528873 5461 ts_tablet_manager.cc:1975] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:26.529016 5461 log.cc:1199] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-2/wal/wals/65bd6fcaed5b47c3a2478d375ace9636
I20261118 02:55:26.529417 5461 ts_tablet_manager.cc:1996] T 65bd6fcaed5b47c3a2478d375ace9636 P 3f1973986ec1466eab5bec01dd81654f: Deleting consensus metadata
I20261118 02:55:26.529667 5467 tablet_replica.cc:333] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:26.529891 5467 raft_consensus.cc:2261] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:26.530104 5467 raft_consensus.cc:2290] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:26.531332 5469 tablet_bootstrap.cc:492] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:26.531466 5469 ts_tablet_manager.cc:1449] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20261118 02:55:26.532280 5469 raft_consensus.cc:360] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.532459 5469 raft_consensus.cc:386] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.532531 5469 raft_consensus.cc:748] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.532830 5470 consensus_queue.cc:261] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.536119 5470 ts_tablet_manager.cc:1480] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20261118 02:55:26.537688 5469 consensus_queue.cc:261] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.540684 5462 ts_tablet_manager.cc:1975] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:26.540812 5462 log.cc:1199] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-1/wal/wals/65bd6fcaed5b47c3a2478d375ace9636
I20261118 02:55:26.541198 5462 ts_tablet_manager.cc:1996] T 65bd6fcaed5b47c3a2478d375ace9636 P 99a39b1d9d814deeb3a8a644e9b392ef: Deleting consensus metadata
I20261118 02:55:26.552300 5470 tablet_bootstrap.cc:492] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:26.554435 5470 tablet_bootstrap.cc:654] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.558333 5469 ts_tablet_manager.cc:1480] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20261118 02:55:26.559616 5469 tablet_bootstrap.cc:492] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:26.562727 5469 tablet_bootstrap.cc:654] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.564500 5468 tablet_bootstrap.cc:492] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:26.564653 5468 ts_tablet_manager.cc:1449] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.013s
I20261118 02:55:26.565452 5468 raft_consensus.cc:360] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.565642 5468 raft_consensus.cc:386] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.565723 5468 raft_consensus.cc:748] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.565989 5468 consensus_queue.cc:261] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.566454 3663 heartbeater.cc:500] Master 127.2.229.126:38025 was elected leader, sending a full tablet report...
I20261118 02:55:26.568331 5468 ts_tablet_manager.cc:1480] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 02:55:26.571219 5469 tablet_bootstrap.cc:492] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:26.571380 5469 ts_tablet_manager.cc:1449] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20261118 02:55:26.572203 5469 raft_consensus.cc:360] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.572392 5469 raft_consensus.cc:386] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.572455 5469 raft_consensus.cc:748] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.572738 5469 consensus_queue.cc:261] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.573143 5470 tablet_bootstrap.cc:492] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:26.573309 5470 ts_tablet_manager.cc:1449] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.003s
I20261118 02:55:26.575390 5469 ts_tablet_manager.cc:1480] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 02:55:26.574066 5470 raft_consensus.cc:360] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.576234 5470 raft_consensus.cc:386] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.576318 5470 raft_consensus.cc:748] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.576576 5470 consensus_queue.cc:261] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.577075 5470 ts_tablet_manager.cc:1480] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20261118 02:55:26.577852 5469 tablet_bootstrap.cc:492] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:26.580029 5469 tablet_bootstrap.cc:654] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.580224 5223 raft_consensus.cc:499] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:26.580389 5223 raft_consensus.cc:521] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.581095 5223 leader_election.cc:290] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:26.581457 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b119ead4bb6548b4a37da02ed6daa6a8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:26.581952 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b119ead4bb6548b4a37da02ed6daa6a8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:26.582132 5470 tablet_bootstrap.cc:492] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:26.582348 3751 raft_consensus.cc:2490] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
W20261118 02:55:26.582592 3553 leader_election.cc:343] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Illegal state: must be running to vote when last-logged opid is not known
I20261118 02:55:26.583514 3552 leader_election.cc:304] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 3f1973986ec1466eab5bec01dd81654f
I20261118 02:55:26.584192 5470 tablet_bootstrap.cc:654] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.585340 5223 raft_consensus.cc:2826] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:26.585444 5223 raft_consensus.cc:499] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:26.585517 5223 raft_consensus.cc:3096] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.588392 5223 raft_consensus.cc:521] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.590905 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b119ead4bb6548b4a37da02ed6daa6a8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:26.591154 5468 tablet_bootstrap.cc:492] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
W20261118 02:55:26.592422 3553 leader_election.cc:343] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Illegal state: must be running to vote when last-logged opid is not known
I20261118 02:55:26.592964 5468 tablet_bootstrap.cc:654] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.595716 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b119ead4bb6548b4a37da02ed6daa6a8" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:26.595944 3751 raft_consensus.cc:3096] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.597872 5468 tablet_bootstrap.cc:492] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:26.598038 5468 ts_tablet_manager.cc:1449] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20261118 02:55:26.598819 5468 raft_consensus.cc:360] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.599017 5468 raft_consensus.cc:386] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.599088 5468 raft_consensus.cc:748] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.599347 5468 consensus_queue.cc:261] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.599607 3751 raft_consensus.cc:2490] T b119ead4bb6548b4a37da02ed6daa6a8 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:26.600179 5223 leader_election.cc:290] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:26.600370 3552 leader_election.cc:304] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters: 3f1973986ec1466eab5bec01dd81654f
I20261118 02:55:26.601939 5468 ts_tablet_manager.cc:1480] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20261118 02:55:26.602519 5467 ts_tablet_manager.cc:1962] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:26.603873 5223 raft_consensus.cc:2826] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:26.604066 5223 raft_consensus.cc:705] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:26.604453 5223 consensus_queue.cc:238] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.603153 5468 tablet_bootstrap.cc:492] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:26.609411 5471 consensus_queue.cc:1049] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:26.611416 5468 tablet_bootstrap.cc:654] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
W20261118 02:55:26.611650 3553 consensus_peers.cc:597] T b119ead4bb6548b4a37da02ed6daa6a8 P 1fdf7d7ece324fefb3a0c376add5b7e8 -> Peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Couldn't send request to peer 3f1973986ec1466eab5bec01dd81654f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:26.619196 5467 ts_tablet_manager.cc:1975] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:26.619350 5467 log.cc:1199] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/65bd6fcaed5b47c3a2478d375ace9636
I20261118 02:55:26.622983 5469 tablet_bootstrap.cc:492] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:26.623145 5469 ts_tablet_manager.cc:1449] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20261118 02:55:26.623934 5469 raft_consensus.cc:360] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.624138 5469 raft_consensus.cc:386] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.624231 5469 raft_consensus.cc:748] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.624473 5469 consensus_queue.cc:261] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.625037 5469 ts_tablet_manager.cc:1480] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:26.625331 5469 tablet_bootstrap.cc:492] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:26.627117 5469 tablet_bootstrap.cc:654] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.628249 5467 ts_tablet_manager.cc:1996] T 65bd6fcaed5b47c3a2478d375ace9636 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:26.630618 5468 tablet_bootstrap.cc:492] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:26.630762 5468 ts_tablet_manager.cc:1449] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20261118 02:55:26.631496 5468 raft_consensus.cc:360] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.631680 5468 raft_consensus.cc:386] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.631759 5468 raft_consensus.cc:748] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.632010 5468 consensus_queue.cc:261] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.632588 5468 ts_tablet_manager.cc:1480] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:26.635655 5468 tablet_bootstrap.cc:492] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:26.637357 5468 tablet_bootstrap.cc:654] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.649585 5470 tablet_bootstrap.cc:492] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:26.649569 5469 tablet_bootstrap.cc:492] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:26.649730 5470 ts_tablet_manager.cc:1449] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.068s user 0.014s sys 0.025s
I20261118 02:55:26.649747 5469 ts_tablet_manager.cc:1449] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.005s
I20261118 02:55:26.650496 5469 raft_consensus.cc:360] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.650521 5470 raft_consensus.cc:360] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.650682 5470 raft_consensus.cc:386] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.650699 5469 raft_consensus.cc:386] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.650760 5469 raft_consensus.cc:748] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.650763 5470 raft_consensus.cc:748] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.651127 5469 consensus_queue.cc:261] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.651186 5470 consensus_queue.cc:261] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.653043 5468 tablet_bootstrap.cc:492] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:26.653216 5468 ts_tablet_manager.cc:1449] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20261118 02:55:26.653954 5468 raft_consensus.cc:360] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.654140 5468 raft_consensus.cc:386] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.654170 5469 ts_tablet_manager.cc:1480] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 02:55:26.654202 5468 raft_consensus.cc:748] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.654500 5468 consensus_queue.cc:261] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.654989 5468 ts_tablet_manager.cc:1480] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 02:55:26.655587 5469 tablet_bootstrap.cc:492] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f: Bootstrap starting.
I20261118 02:55:26.658234 5468 tablet_bootstrap.cc:492] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8: Bootstrap starting.
I20261118 02:55:26.657541 5470 ts_tablet_manager.cc:1480] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20261118 02:55:26.660919 5468 tablet_bootstrap.cc:654] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.663338 5470 tablet_bootstrap.cc:492] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:26.664065 5469 tablet_bootstrap.cc:654] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.673298 5470 tablet_bootstrap.cc:654] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.677798 5469 tablet_bootstrap.cc:492] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f: No bootstrap required, opened a new log
I20261118 02:55:26.677976 5469 ts_tablet_manager.cc:1449] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.005s
I20261118 02:55:26.678954 5469 raft_consensus.cc:360] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.679150 5469 raft_consensus.cc:386] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.679239 5469 raft_consensus.cc:748] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.679417 5470 tablet_bootstrap.cc:492] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:26.679479 5469 consensus_queue.cc:261] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.679586 5470 ts_tablet_manager.cc:1449] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20261118 02:55:26.680831 5468 tablet_bootstrap.cc:492] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8: No bootstrap required, opened a new log
I20261118 02:55:26.680965 5468 ts_tablet_manager.cc:1449] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.010s
I20261118 02:55:26.681173 5469 ts_tablet_manager.cc:1480] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20261118 02:55:26.681653 5468 raft_consensus.cc:360] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.681818 5468 raft_consensus.cc:386] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.681892 5468 raft_consensus.cc:748] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.682098 5468 consensus_queue.cc:261] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.683750 5468 ts_tablet_manager.cc:1480] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20261118 02:55:26.685869 5470 raft_consensus.cc:360] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.686022 5470 raft_consensus.cc:386] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.686065 5470 raft_consensus.cc:748] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.686266 5470 consensus_queue.cc:261] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.689347 5470 ts_tablet_manager.cc:1480] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20261118 02:55:26.689688 5470 tablet_bootstrap.cc:492] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef: Bootstrap starting.
I20261118 02:55:26.691386 5424 raft_consensus.cc:499] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:26.691532 5424 raft_consensus.cc:521] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.692039 5470 tablet_bootstrap.cc:654] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef: Neither blocks nor log segments found. Creating new log.
I20261118 02:55:26.692778 5424 leader_election.cc:290] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:26.693562 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b400925bbe6f4639b3d9e76cbb1ca681" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:26.693739 3751 raft_consensus.cc:2490] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:26.694175 3836 leader_election.cc:304] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:26.694469 5424 raft_consensus.cc:2826] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:26.694566 5424 raft_consensus.cc:499] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:26.694530 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b400925bbe6f4639b3d9e76cbb1ca681" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:26.694648 5424 raft_consensus.cc:3096] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.694712 3616 raft_consensus.cc:2490] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 0.
I20261118 02:55:26.695223 5470 tablet_bootstrap.cc:492] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef: No bootstrap required, opened a new log
I20261118 02:55:26.695348 5470 ts_tablet_manager.cc:1449] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20261118 02:55:26.696225 5470 raft_consensus.cc:360] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.696408 5470 raft_consensus.cc:386] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 02:55:26.696491 5470 raft_consensus.cc:748] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Initialized, Role: FOLLOWER
I20261118 02:55:26.697541 5424 raft_consensus.cc:521] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.698520 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b400925bbe6f4639b3d9e76cbb1ca681" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:26.701494 3616 raft_consensus.cc:3096] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.698891 5424 leader_election.cc:290] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:26.696735 5470 consensus_queue.cc:261] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.698750 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b400925bbe6f4639b3d9e76cbb1ca681" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:26.702953 3751 raft_consensus.cc:3096] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.703986 5470 ts_tablet_manager.cc:1480] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20261118 02:55:26.704284 3616 raft_consensus.cc:2490] T b400925bbe6f4639b3d9e76cbb1ca681 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:26.704876 3838 leader_election.cc:304] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:26.705199 5424 raft_consensus.cc:2826] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:26.705335 5424 raft_consensus.cc:705] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [term 1 LEADER]: Becoming Leader. State: Replica: 3f1973986ec1466eab5bec01dd81654f, State: Running, Role: LEADER
I20261118 02:55:26.705647 5424 consensus_queue.cc:238] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.706207 3751 raft_consensus.cc:2490] T b400925bbe6f4639b3d9e76cbb1ca681 P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1973986ec1466eab5bec01dd81654f in term 1.
I20261118 02:55:26.710197 5424 consensus_queue.cc:1049] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:26.710708 5465 consensus_queue.cc:1049] T b400925bbe6f4639b3d9e76cbb1ca681 P 3f1973986ec1466eab5bec01dd81654f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 02:55:26.772207 3690 connection.cc:573] server connection from 127.0.0.1:50108 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:26.772219 3837 connection.cc:573] client connection to 127.2.229.126:38025 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:26.772219 3838 connection.cc:573] server connection from 127.0.0.1:35078 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:26.772461 3690 connection.cc:573] client connection to 127.2.229.126:38025 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:26.772687 3956 connection.cc:573] client connection to 127.2.229.126:38025 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:26.772970 3553 connection.cc:573] client connection to 127.2.229.126:38025 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:26.773291 3553 connection.cc:573] server connection from 127.0.0.1:46360 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:26.773315 3956 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.229.126:38025)
W20261118 02:55:26.773818 3956 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.229.126:38025)
W20261118 02:55:26.774013 3956 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.229.126:38025)
W20261118 02:55:26.774312 3956 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.229.126:38025)
W20261118 02:55:26.774842 5464 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:26.773156 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.2.229.126:38025 (local address 127.2.229.67:54969)
1118 02:55:26.774066 (+ 910us) negotiation.cc:107] Waiting for socket to connect
1118 02:55:26.774087 (+ 21us) client_negotiation.cc:175] Beginning negotiation
1118 02:55:26.774217 (+ 130us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 02:55:26.774803 (+ 586us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":783}
W20261118 02:55:26.775566 5431 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:26.774633 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.2.229.126:38025 (local address 127.2.229.66:38331)
1118 02:55:26.774905 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1118 02:55:26.774922 (+ 17us) client_negotiation.cc:175] Beginning negotiation
1118 02:55:26.775038 (+ 116us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 02:55:26.775526 (+ 488us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":175}
I20261118 02:55:26.789143 5223 raft_consensus.cc:499] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:26.789335 5223 raft_consensus.cc:521] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.789904 5223 leader_election.cc:290] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:26.790370 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c46a0eff7633462fb3719143c1f7cfbb" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:26.790398 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c46a0eff7633462fb3719143c1f7cfbb" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:26.790587 3901 raft_consensus.cc:2490] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:26.790592 3751 raft_consensus.cc:2490] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:26.790972 3553 leader_election.cc:304] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:26.791229 5223 raft_consensus.cc:2826] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:26.791308 5223 raft_consensus.cc:499] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:26.791404 5223 raft_consensus.cc:3096] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.794036 5223 raft_consensus.cc:521] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.794605 5223 leader_election.cc:290] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:26.794925 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c46a0eff7633462fb3719143c1f7cfbb" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:26.795081 3901 raft_consensus.cc:3096] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.795058 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c46a0eff7633462fb3719143c1f7cfbb" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:26.795241 3751 raft_consensus.cc:3096] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.797178 3901 raft_consensus.cc:2490] T c46a0eff7633462fb3719143c1f7cfbb P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:26.797281 3751 raft_consensus.cc:2490] T c46a0eff7633462fb3719143c1f7cfbb P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:26.797603 3553 leader_election.cc:304] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:26.797863 5223 raft_consensus.cc:2826] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:26.798029 5223 raft_consensus.cc:705] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:26.798347 5223 consensus_queue.cc:238] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.805913 5223 consensus_queue.cc:1049] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:26.806227 5491 consensus_queue.cc:1049] T c46a0eff7633462fb3719143c1f7cfbb P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:26.819752 5471 raft_consensus.cc:499] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:26.819947 5471 raft_consensus.cc:521] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.820597 5471 leader_election.cc:290] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:26.821031 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d8766c4d56fa4689a69b153ea7a613ee" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:26.821228 3751 raft_consensus.cc:2490] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:26.821592 3552 leader_election.cc:304] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:26.821857 5471 raft_consensus.cc:2826] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:26.821936 5471 raft_consensus.cc:499] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:26.822029 5471 raft_consensus.cc:3096] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.823872 5471 raft_consensus.cc:521] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.824396 5471 leader_election.cc:290] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:26.824501 3901 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d8766c4d56fa4689a69b153ea7a613ee" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:26.824764 3901 raft_consensus.cc:2490] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 0.
I20261118 02:55:26.825029 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d8766c4d56fa4689a69b153ea7a613ee" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:26.825215 3902 raft_consensus.cc:3096] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.825284 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d8766c4d56fa4689a69b153ea7a613ee" candidate_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef"
I20261118 02:55:26.825453 3751 raft_consensus.cc:3096] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:26.827188 3902 raft_consensus.cc:2490] T d8766c4d56fa4689a69b153ea7a613ee P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:26.827325 3751 raft_consensus.cc:2490] T d8766c4d56fa4689a69b153ea7a613ee P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fdf7d7ece324fefb3a0c376add5b7e8 in term 1.
I20261118 02:55:26.827428 5476 raft_consensus.cc:499] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:26.827628 3553 leader_election.cc:304] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 3f1973986ec1466eab5bec01dd81654f; no voters:
I20261118 02:55:26.827564 5476 raft_consensus.cc:521] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:26.827956 5471 raft_consensus.cc:2826] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:26.828289 5476 leader_election.cc:290] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:26.828122 5471 raft_consensus.cc:705] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf7d7ece324fefb3a0c376add5b7e8, State: Running, Role: LEADER
I20261118 02:55:26.828703 5471 consensus_queue.cc:238] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:26.828722 3751 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b119ead4bb6548b4a37da02ed6daa6a8" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" is_pre_election: true
I20261118 02:55:26.831491 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b119ead4bb6548b4a37da02ed6daa6a8" candidate_uuid: "3f1973986ec1466eab5bec01dd81654f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:26.832070 3838 leader_election.cc:304] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f1973986ec1466eab5bec01dd81654f; no voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:26.832361 5476 raft_consensus.cc:2771] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 02:55:26.832858 5223 consensus_queue.cc:1049] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:26.833478 5223 consensus_queue.cc:1049] T d8766c4d56fa4689a69b153ea7a613ee P 1fdf7d7ece324fefb3a0c376add5b7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:26.936182 5518 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:55:26.936419 5518 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:26.937175 5518 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:26.939237 5342 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:26.939416 5342 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:26.939498 5342 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:26.939752 5518 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20261118 02:55:27.056810 5508 raft_consensus.cc:499] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 02:55:27.057022 5508 raft_consensus.cc:521] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:27.057701 5508 leader_election.cc:290] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:27.058296 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c1f0147a1efb4f4f813478f119043bec" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" is_pre_election: true
I20261118 02:55:27.058320 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c1f0147a1efb4f4f813478f119043bec" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f" is_pre_election: true
I20261118 02:55:27.058497 3616 raft_consensus.cc:2490] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:27.058538 3902 raft_consensus.cc:2490] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 0.
I20261118 02:55:27.058902 3691 leader_election.cc:304] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:27.059204 5508 raft_consensus.cc:2826] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 02:55:27.059333 5508 raft_consensus.cc:499] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 02:55:27.059420 5508 raft_consensus.cc:3096] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:27.062515 5508 raft_consensus.cc:521] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:27.063611 5508 leader_election.cc:290] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Requested vote from peers 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:27.063746 3616 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c1f0147a1efb4f4f813478f119043bec" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8"
I20261118 02:55:27.063983 3616 raft_consensus.cc:3096] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:27.064313 3902 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c1f0147a1efb4f4f813478f119043bec" candidate_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1973986ec1466eab5bec01dd81654f"
I20261118 02:55:27.064481 3902 raft_consensus.cc:3096] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
I20261118 02:55:27.066200 3616 raft_consensus.cc:2490] T c1f0147a1efb4f4f813478f119043bec P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:27.066627 3691 leader_election.cc:304] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8, 99a39b1d9d814deeb3a8a644e9b392ef; no voters:
I20261118 02:55:27.066928 5508 raft_consensus.cc:2826] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 FOLLOWER]: Leader election won for term 1
I20261118 02:55:27.067077 5508 raft_consensus.cc:705] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [term 1 LEADER]: Becoming Leader. State: Replica: 99a39b1d9d814deeb3a8a644e9b392ef, State: Running, Role: LEADER
I20261118 02:55:27.067217 3902 raft_consensus.cc:2490] T c1f0147a1efb4f4f813478f119043bec P 3f1973986ec1466eab5bec01dd81654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a39b1d9d814deeb3a8a644e9b392ef in term 1.
I20261118 02:55:27.067411 5508 consensus_queue.cc:238] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } }
I20261118 02:55:27.070578 5508 consensus_queue.cc:1049] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:27.070902 5507 consensus_queue.cc:1049] T c1f0147a1efb4f4f813478f119043bec P 99a39b1d9d814deeb3a8a644e9b392ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 02:55:27.478394 2965 external_mini_cluster.cc:1372] Running /tmp/dist-test-task8fD3U4/build/asan/bin/kudu
/tmp/dist-test-task8fD3U4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.124:36695
--webserver_interface=127.2.229.124
--webserver_port=39557
--builtin_ntp_servers=127.2.229.84:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 02:55:27.614560 5531 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:55:27.614765 5531 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:27.615352 5531 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
W20261118 02:55:27.617887 5342 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:27.618136 5342 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695): Network error: Client connection negotiation failed: client connection to 127.2.229.124:36695: connect: Connection refused (error 111)
I20261118 02:55:27.618258 5342 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 813a6937cea442ef81645fccc32207f4
I20261118 02:55:27.618496 5531 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20261118 02:55:27.684228 3902 raft_consensus.cc:3096] T b119ead4bb6548b4a37da02ed6daa6a8 P 3f1973986ec1466eab5bec01dd81654f [term 0 FOLLOWER]: Advancing to term 1
W20261118 02:55:27.833762 5526 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 02:55:27.834057 5526 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 02:55:27.834132 5526 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 02:55:27.843071 5526 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 02:55:27.843153 5526 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 02:55:27.843206 5526 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 02:55:27.843257 5526 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 02:55:27.853453 5526 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.229.84:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39081
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.229.124:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.229.124
--webserver_port=39557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 02:43:21 UTC on bdcb31816ec0
build id 14837
ASAN enabled
I20261118 02:55:27.855036 5526 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 02:55:27.856874 5526 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 02:55:27.865374 5544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:27.865410 5545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 02:55:27.865906 5547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 02:55:27.866293 5526 server_base.cc:1061] running on GCE node
I20261118 02:55:27.867000 5526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 02:55:27.868415 5526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 02:55:27.869627 5526 hybrid_clock.cc:648] HybridClock initialized: now 1794970527869583 us; error 26 us; skew 500 ppm
I20261118 02:55:27.869941 5526 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 02:55:27.872119 5526 webserver.cc:577] Webserver started at http://127.2.229.124:39557/ using document root <none> and password file <none>
I20261118 02:55:27.872843 5526 fs_manager.cc:362] Metadata directory not provided
I20261118 02:55:27.872949 5526 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 02:55:27.878437 5526 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20261118 02:55:27.881456 5553 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 02:55:27.882910 5526 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20261118 02:55:27.883075 5526 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data,/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
uuid: "813a6937cea442ef81645fccc32207f4"
format_stamp: "Formatted at 2026-11-18 02:54:57 on dist-test-slave-49p1"
I20261118 02:55:27.883770 5526 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 02:55:27.903831 5526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 02:55:27.904734 5526 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 02:55:27.905011 5526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 02:55:27.923594 5526 rpc_server.cc:307] RPC server started. Bound to: 127.2.229.124:36695
I20261118 02:55:27.923611 5604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.229.124:36695 every 8 connection(s)
I20261118 02:55:27.925367 5526 server_base.cc:1193] Dumped server information to /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/master-2/data/info.pb
I20261118 02:55:27.926820 2965 external_mini_cluster.cc:1434] Started /tmp/dist-test-task8fD3U4/build/asan/bin/kudu as pid 5526
I20261118 02:55:27.934008 5605 sys_catalog.cc:263] Verifying existing consensus state
I20261118 02:55:27.937644 5605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap starting.
I20261118 02:55:27.959455 5605 log.cc:826] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Log is configured to *not* fsync() on all Append() calls
I20261118 02:55:28.042068 5605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=113 overwritten=0 applied=113 ignored=0} inserts{seen=75 ignored=0} mutations{seen=100 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 02:55:28.042614 5605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: Bootstrap complete.
I20261118 02:55:28.049194 5605 raft_consensus.cc:360] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:28.049918 5605 raft_consensus.cc:748] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813a6937cea442ef81645fccc32207f4, State: Initialized, Role: FOLLOWER
I20261118 02:55:28.050611 5605 consensus_queue.cc:261] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 113, Last appended: 8.113, Last appended by leader: 113, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:28.051461 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:28.051775 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:28.052516 5605 sys_catalog.cc:565] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 02:55:28.072705 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:55:28.078828 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:28.081604 5624 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:28.081689 5624 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: loading cluster ID for follower catalog manager: success
I20261118 02:55:28.085057 5624 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: acquiring CA information for follower catalog manager: success
I20261118 02:55:28.087491 5624 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 02:55:28.097249 3664 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:55:28.097417 3664 heartbeater.cc:500] Master 127.2.229.124:36695 was elected leader, sending a full tablet report...
I20261118 02:55:28.101508 5569 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" instance_seqno: 1794970498532688) as {username='slave'} at 127.2.229.65:45763; Asking this server to re-register.
I20261118 02:55:28.102363 3664 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:28.104434 5569 ts_manager.cc:210] Registered new tserver with Master: 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705)
I20261118 02:55:28.136616 5491 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:55:28.136852 5491 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:28.137568 5491 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:28.138643 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:55:28.138813 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:28.138906 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:28.139163 5491 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:28.197311 5630 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:55:28.197511 5630 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:28.198184 5630 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:28.203583 3801 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
I20261118 02:55:28.210520 5570 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" instance_seqno: 1794970499028671) as {username='slave'} at 127.2.229.66:36123; Asking this server to re-register.
I20261118 02:55:28.211706 3801 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:28.213747 5570 ts_manager.cc:210] Registered new tserver with Master: 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377)
I20261118 02:55:28.219092 3948 heartbeater.cc:345] Connected to a master server at 127.2.229.124:36695
W20261118 02:55:28.219417 5342 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:28.221925 5569 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" instance_seqno: 1794970499527172) as {username='slave'} at 127.2.229.67:59991; Asking this server to re-register.
I20261118 02:55:28.222345 5580 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 10 candidate_status { last_received { term: 9 index: 117 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4" is_pre_election: true
I20261118 02:55:28.222728 3948 heartbeater.cc:462] Registering TS with master...
I20261118 02:55:28.222995 5580 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 457eb38cba564ecd9f6f11245172260b in term 8.
I20261118 02:55:28.223645 5342 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d
I20261118 02:55:28.223961 5630 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20261118 02:55:28.224077 5630 raft_consensus.cc:499] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 26c6b20fa203404a88463ea27bdd3c7d)
I20261118 02:55:28.224187 5630 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 9 FOLLOWER]: Advancing to term 10
I20261118 02:55:28.224674 5569 ts_manager.cc:210] Registered new tserver with Master: 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:28.226445 5630 raft_consensus.cc:521] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:28.227042 5630 leader_election.cc:290] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 election: Requested vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 813a6937cea442ef81645fccc32207f4 (127.2.229.124:36695)
I20261118 02:55:28.227790 5580 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "457eb38cba564ecd9f6f11245172260b" candidate_term: 10 candidate_status { last_received { term: 9 index: 117 } } ignore_live_leader: false dest_uuid: "813a6937cea442ef81645fccc32207f4"
I20261118 02:55:28.227967 5580 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 8 FOLLOWER]: Advancing to term 10
I20261118 02:55:28.232302 5580 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 457eb38cba564ecd9f6f11245172260b in term 10.
I20261118 02:55:28.232748 5342 leader_election.cc:304] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 457eb38cba564ecd9f6f11245172260b, 813a6937cea442ef81645fccc32207f4; no voters:
I20261118 02:55:28.233035 5630 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 10 FOLLOWER]: Leader election won for term 10
W20261118 02:55:28.233604 5342 leader_election.cc:336] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:28.236358 5630 raft_consensus.cc:705] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [term 10 LEADER]: Becoming Leader. State: Replica: 457eb38cba564ecd9f6f11245172260b, State: Running, Role: LEADER
I20261118 02:55:28.236807 5630 consensus_queue.cc:238] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 9.117, Last appended by leader: 117, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:28.238528 5636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 10 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:28.241077 5580 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Refusing update from remote peer 457eb38cba564ecd9f6f11245172260b: Log matching property violated. Preceding OpId in replica: term: 8 index: 113. Preceding OpId from leader: term: 10 index: 118. (index mismatch)
I20261118 02:55:28.243892 5638 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [LEADER]: Connected to new peer: Peer: permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 113, Time since last communication: 0.000s
I20261118 02:55:28.249001 5636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
W20261118 02:55:28.249681 5342 consensus_peers.cc:597] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b -> Peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Couldn't send request to peer 26c6b20fa203404a88463ea27bdd3c7d. Status: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:28.254060 5642 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 02:55:28.257992 5642 catalog_manager.cc:694] Loaded metadata for table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190]
I20261118 02:55:28.258582 5642 catalog_manager.cc:694] Loaded metadata for table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c]
I20261118 02:55:28.259119 5641 mvcc.cc:204] Tried to move back new op lower bound from 7352199276076613632 to 7352199275831820288. Current Snapshot: MvccSnapshot[applied={T|T < 7352199276076613632 or (T in {7352199276076613632})}]
I20261118 02:55:28.259443 5642 catalog_manager.cc:694] Loaded metadata for table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6]
I20261118 02:55:28.259922 5642 catalog_manager.cc:694] Loaded metadata for table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73]
I20261118 02:55:28.260382 5642 catalog_manager.cc:694] Loaded metadata for table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e]
I20261118 02:55:28.260816 5642 catalog_manager.cc:694] Loaded metadata for table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b]
I20261118 02:55:28.260901 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457eb38cba564ecd9f6f11245172260b. Latest consensus state: current_term: 10 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:28.261139 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:28.261464 5642 catalog_manager.cc:694] Loaded metadata for table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da]
I20261118 02:55:28.261788 5642 catalog_manager.cc:694] Loaded metadata for table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66]
I20261118 02:55:28.262087 5642 catalog_manager.cc:694] Loaded metadata for table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d]
I20261118 02:55:28.262394 5642 catalog_manager.cc:694] Loaded metadata for table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db]
I20261118 02:55:28.262754 5642 catalog_manager.cc:694] Loaded metadata for table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf]
I20261118 02:55:28.262694 5644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:28.262990 5644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:28.263064 5642 catalog_manager.cc:694] Loaded metadata for table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b]
I20261118 02:55:28.263397 5642 catalog_manager.cc:694] Loaded metadata for table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d]
I20261118 02:55:28.263708 5642 catalog_manager.cc:694] Loaded metadata for table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7]
I20261118 02:55:28.264014 5642 catalog_manager.cc:694] Loaded metadata for table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67]
I20261118 02:55:28.264436 5642 catalog_manager.cc:694] Loaded metadata for table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d]
I20261118 02:55:28.264829 5642 catalog_manager.cc:694] Loaded metadata for table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec]
I20261118 02:55:28.265306 5642 catalog_manager.cc:694] Loaded metadata for table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1]
I20261118 02:55:28.265707 5642 catalog_manager.cc:694] Loaded metadata for table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f]
I20261118 02:55:28.266211 5642 catalog_manager.cc:694] Loaded metadata for table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139]
I20261118 02:55:28.266783 5644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:28.266963 5644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 02:55:28.269320 5642 tablet_loader.cc:96] loaded metadata for tablet 0e901a03b90e4c978eb98be87163cfe5 (table default.table_d4fec0aaed15433ea373740872db53d9 [id=00713c8ca8db4bf9a860d32614f99190])
I20261118 02:55:28.269738 5642 tablet_loader.cc:96] loaded metadata for tablet 1904140da78b4567a9decb250c6f74a3 (table default.table_61b6eaac749b4be0b0ddfba6ff9aa706 [id=e0ed3867fcbb42a4b0133f38e0372a6f])
I20261118 02:55:28.270381 5642 tablet_loader.cc:96] loaded metadata for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc (table default.table_a4f08da750c54c49b9b7d73ebcf52624 [id=790b0571767b49eead05512b819d27db])
I20261118 02:55:28.270735 5642 tablet_loader.cc:96] loaded metadata for tablet 2c92955aa65142f1bd20f4b8f3321500 (table default.table_38304cbb980440c2a8b8a6c13f49949c [id=7140b942170b4483b0e01e32d5ba5e6d])
I20261118 02:55:28.270915 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "457eb38cba564ecd9f6f11245172260b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } } }
I20261118 02:55:28.271173 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b [sys.catalog]: This master's current role is: LEADER
I20261118 02:55:28.271363 5642 tablet_loader.cc:96] loaded metadata for tablet 462a9be0ae114b01a9c69416185803f8 (table default.table_571d1360811c42dc815efb439b2b52e6 [id=e797fad7075b40d9bdd674a0e0023139])
I20261118 02:55:28.271699 5642 tablet_loader.cc:96] loaded metadata for tablet 46f05790dbdc4b8185a7b0362f604cf8 (table default.table_28c62f411da74545aa42dad7325bc064 [id=7d99cc94d0ab4bf9a21881fc6dcf91bf])
I20261118 02:55:28.273630 5642 tablet_loader.cc:96] loaded metadata for tablet 72f427c72a554ac1a0a2f2c3a78865da (table default.table_30f082fbe850451b80d98df75ed1081a [id=44b390866720455b8d2714eae004383b])
I20261118 02:55:28.273978 5642 tablet_loader.cc:96] loaded metadata for tablet 78189cccd457466bb5d01231038fec24 (table default.table_a7f3710d39fb447da94e16d4ab3933fd [id=4a09f210503b499f8e842f6daafb33da])
I20261118 02:55:28.274513 5642 tablet_loader.cc:96] loaded metadata for tablet 8bed657d0da64d95851f8a86216b6c2c (table default.table_6e09d0e89581458183ed524cc1404849 [id=01b05aad48954b8fb2aa6d7b31a6578c])
I20261118 02:55:28.275988 5642 tablet_loader.cc:96] loaded metadata for tablet a67bc60c7ecb4b7e8e03e04267bb91a5 (table default.table_0385cef6593c419eb426657c8cbef2bf [id=8a5ea4b95bee4db7af9ba875250d564b])
I20261118 02:55:28.276566 5642 tablet_loader.cc:96] loaded metadata for tablet ab61b43e639a4e75bc2e7009ea86a64f (table default.table_f0ae490a0df64e1097e144c7a370df61 [id=a59b9425c01049269883cf7767936c67])
I20261118 02:55:28.276911 5642 tablet_loader.cc:96] loaded metadata for tablet b119ead4bb6548b4a37da02ed6daa6a8 (table default.table_287099a68a3540f1be0284c71d4d7056 [id=4e5b6eaa70d34daf8a68fc17ec1afb66])
I20261118 02:55:28.277233 5642 tablet_loader.cc:96] loaded metadata for tablet b129518ce23c45758ebcaa55410757d8 (table default.table_30b09a2798a54b01be7618b8a28eafaf [id=dd0f01caa36a4d3e9795daaf698b1ce1])
I20261118 02:55:28.277590 5642 tablet_loader.cc:96] loaded metadata for tablet b400925bbe6f4639b3d9e76cbb1ca681 (table default.table_4f7796d5d5d54def843b971e66d15ea6 [id=a2ece7af36934cdf80ad8458024adec7])
I20261118 02:55:28.278465 5642 tablet_loader.cc:96] loaded metadata for tablet c1f0147a1efb4f4f813478f119043bec (table default.table_a14cb6414555446286bf35bc5c6fb910 [id=0ce698410f004edaa9ad4ba1afe2de73])
I20261118 02:55:28.278752 5642 tablet_loader.cc:96] loaded metadata for tablet c46a0eff7633462fb3719143c1f7cfbb (table default.table_22f31dd1ff9e4185a6618f11b5493434 [id=aea45564ce10469393d3c7cc03b0bb9d])
I20261118 02:55:28.279328 5642 tablet_loader.cc:96] loaded metadata for tablet ccf78258c9634540a178855a142f4d4b (table default.table_2e3917ff98f2474984580e61caa9c17b [id=2584489d44ac4c6b90ca7c9f390c617e])
I20261118 02:55:28.279902 5642 tablet_loader.cc:96] loaded metadata for tablet d8766c4d56fa4689a69b153ea7a613ee (table default.table_87d592a09c3348028719a603956fbce6 [id=9724ed64fa5043bcbfee1caefac8e86d])
I20261118 02:55:28.280303 5642 tablet_loader.cc:96] loaded metadata for tablet e71c5cd71ebf4fa6bd158d59ed2b191d (table default.table_8ae376d4d4d64414b5c08dc3c2e58bc4 [id=052f3134115244a6a2443e96121313e6])
I20261118 02:55:28.281126 5642 tablet_loader.cc:96] loaded metadata for tablet fbdc4c7b574e4f94b816fb0c0b14c185 (table default.table_c113c2b57e17479e92e241234a241af7 [id=da0cb25ccfed426ca61f0dd9f93d30ec])
I20261118 02:55:28.281781 5642 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 02:55:28.283506 5642 catalog_manager.cc:1310] Loaded cluster ID: 4547b4c2697449c8bc7f957ed454ac1e
I20261118 02:55:28.283587 5642 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 02:55:28.285229 5642 catalog_manager.cc:1555] Loading token signing keys...
I20261118 02:55:28.286448 5642 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Loaded TSK: 0
I20261118 02:55:28.287295 5642 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 02:55:28.288455 5491 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:55:28.288620 5642 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 02:55:28.288666 5491 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:28.289443 5491 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
I20261118 02:55:28.290257 5642 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: 20
W20261118 02:55:28.290436 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:55:28.290585 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:55:28.290745 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:28.291020 5491 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:28.344012 5356 master_service.cc:965] ReplaceTablet: received request to replace tablet 0e901a03b90e4c978eb98be87163cfe5 from {username='slave'} at 127.0.0.1:49658
I20261118 02:55:28.344053 5354 master_service.cc:965] ReplaceTablet: received request to replace tablet 2b01d7e596bd4790b8ba5dc88d8e12cc from {username='slave'} at 127.0.0.1:49658
I20261118 02:55:28.347713 5353 master_service.cc:965] ReplaceTablet: received request to replace tablet d8766c4d56fa4689a69b153ea7a613ee from {username='slave'} at 127.0.0.1:49658
I20261118 02:55:28.347838 5350 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49658:
table { table_name: "default.Table_2e3917ff98f2474984580e61caa9c17b" } new_table_name: "default.Table_3c225da1f7fa435083909e053ec9751a" modify_external_catalogs: true
I20261118 02:55:28.347947 5348 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49658:
table { table_name: "default.Table_8ae376d4d4d64414b5c08dc3c2e58bc4" } new_table_name: "default.Table_5091d82298ca44bea5335d4989eb02d8" modify_external_catalogs: true
I20261118 02:55:28.347772 5351 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49658:
table { table_name: "default.Table_87d592a09c3348028719a603956fbce6" } modify_external_catalogs: true
I20261118 02:55:28.348358 5349 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49658:
table { table_name: "default.Table_a14cb6414555446286bf35bc5c6fb910" } new_table_name: "default.Table_324a2600fef947748ef03e27f7cb7e09" modify_external_catalogs: true
I20261118 02:55:28.348433 5351 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49658:
table { table_name: "default.Table_87d592a09c3348028719a603956fbce6" } modify_external_catalogs: true
I20261118 02:55:28.348896 5352 master_service.cc:965] ReplaceTablet: received request to replace tablet 2c92955aa65142f1bd20f4b8f3321500 from {username='slave'} at 127.0.0.1:49658
I20261118 02:55:28.350191 5347 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49658:
name: "default.Table_dc982811b980416484416db1f74c685d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 02:55:28.355518 5355 master_service.cc:965] ReplaceTablet: received request to replace tablet ccf78258c9634540a178855a142f4d4b from {username='slave'} at 127.0.0.1:49658
F20261118 02:55:28.359395 5409 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 21, previous 21; dumping first 2 out of 15 received events: NotificationEvent(eventId=21, eventTime=1794970503, eventType=CREATE_TABLE, dbName=default, tableName=table_f0ae490a0df64e1097e144c7a370df61, message={"server":"","servicePrincipal":"","db":"default","table":"table_f0ae490a0df64e1097e144c7a370df61","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_f0ae490a0df64e1097e144c7a370df61\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970503},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_f0ae490a0df64e1097e144c7a370df61\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_f0ae490a0df64e1097e144c7a370df61\",\"kudu.table_id\":\"a59b9425c01049269883cf7767936c67\",\"transient_lastDdlTime\":\"1794970503\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=21, eventTime=1794970503, eventType=DROP_TABLE, dbName=default, tableName=table_c113c2b57e17479e92e241234a241af7, message={"server":"","servicePrincipal":"","db":"default","table":"table_c113c2b57e17479e92e241234a241af7","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_c113c2b57e17479e92e241234a241af7\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794970500},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/warehouse/table_c113c2b57e17479e92e241234a241af7\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c113c2b57e17479e92e241234a241af7\",\"kudu.table_id\":\"da0cb25ccfed426ca61f0dd9f93d30ec\",\"transient_lastDdlTime\":\"1794970500\",\"comment\":\"\",\"kudu.master_addresses\":\"127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4547b4c2697449c8bc7f957ed454ac1e\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794970503}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20261118 02:55:28.362301 5356 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet 0e901a03b90e4c978eb98be87163cfe5
@ 0x7f93d6efed5d google::LogMessage::Fail() at ??:0
I20261118 02:55:28.364419 5356 catalog_manager.cc:6936] ReplaceTablet: tablet 0e901a03b90e4c978eb98be87163cfe5 deleted and replaced by tablet f17bfd60c79f46efa6b801c3cb892ec0
@ 0x7f93d6f02b23 google::LogMessage::SendToLog() at ??:0
@ 0x7f93d6efe75c google::LogMessage::Flush() at ??:0
I20261118 02:55:28.366576 5356 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49658:
table { table_name: "default.Table_0385cef6593c419eb426657c8cbef2bf" } new_table_name: "default.Table_496441d99faa4e2b8c7cd98e4a5ce745" modify_external_catalogs: true
@ 0x7f93d6efe9a1 google::LogMessage::~LogMessage() at ??:0
I20261118 02:55:28.374794 5354 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet 2b01d7e596bd4790b8ba5dc88d8e12cc
I20261118 02:55:28.375406 5354 catalog_manager.cc:6936] ReplaceTablet: tablet 2b01d7e596bd4790b8ba5dc88d8e12cc deleted and replaced by tablet 96c806f34d0643b68afdae4e9eaf3498
@ 0x7f93ec37e9a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 02:55:28.388564 3597 tablet_service.cc:1547] Processing DeleteTablet for tablet 0e901a03b90e4c978eb98be87163cfe5 with delete_type TABLET_DATA_DELETED (replaced by tablet f17bfd60c79f46efa6b801c3cb892ec0) from {username='slave'} at 127.0.0.1:46376
I20261118 02:55:28.388963 3595 tablet_service.cc:1547] Processing DeleteTablet for tablet 2b01d7e596bd4790b8ba5dc88d8e12cc with delete_type TABLET_DATA_DELETED (replaced by tablet 96c806f34d0643b68afdae4e9eaf3498) from {username='slave'} at 127.0.0.1:46376
@ 0x7f93ec37d251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f93ec384ecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f93ec384d49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
@ 0x7f93d8347dc9 kudu::Thread::SuperviseThread() at ??:0
I20261118 02:55:28.415970 5353 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 457eb38cba564ecd9f6f11245172260b: Sending DeleteTablet for 3 replicas of tablet d8766c4d56fa4689a69b153ea7a613ee
I20261118 02:55:28.416579 5353 catalog_manager.cc:6936] ReplaceTablet: tablet d8766c4d56fa4689a69b153ea7a613ee deleted and replaced by tablet f5511619f39f4deba7ca2cf13ad37258
@ 0x7f93dba226db start_thread at ??:0
@ 0x7f93d449261f clone at ??:0
I20261118 02:55:28.420300 5657 tablet_replica.cc:333] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:28.420519 5657 raft_consensus.cc:2261] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:28.420724 5657 raft_consensus.cc:2290] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:28.421166 5657 ts_tablet_manager.cc:1962] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:28.426870 5657 ts_tablet_manager.cc:1975] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:28.426992 5657 log.cc:1199] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/0e901a03b90e4c978eb98be87163cfe5
I20261118 02:55:28.427407 5657 ts_tablet_manager.cc:1996] T 0e901a03b90e4c978eb98be87163cfe5 P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
I20261118 02:55:28.431715 5657 tablet_replica.cc:333] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: stopping tablet replica
I20261118 02:55:28.432021 5657 raft_consensus.cc:2261] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 02:55:28.432227 5657 raft_consensus.cc:2290] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 02:55:28.432813 5657 ts_tablet_manager.cc:1962] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 02:55:28.439225 5657 ts_tablet_manager.cc:1975] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 02:55:28.439370 5657 log.cc:1199] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting WAL directory at /tmp/dist-test-task8fD3U4/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794970481606018-2965-0/minicluster-data/ts-0/wal/wals/2b01d7e596bd4790b8ba5dc88d8e12cc
I20261118 02:55:28.439713 5657 ts_tablet_manager.cc:1996] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 1fdf7d7ece324fefb3a0c376add5b7e8: Deleting consensus metadata
W20261118 02:55:28.624691 3551 connection.cc:573] server connection from 127.0.0.1:46376 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:28.624712 3552 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:28.624949 5650 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:28.365265 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.2.229.67:42237)
1118 02:55:28.412363 (+ 47098us) server_negotiation.cc:207] Beginning negotiation
1118 02:55:28.412369 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1118 02:55:28.412391 (+ 22us) server_negotiation.cc:408] Connection header received
1118 02:55:28.412463 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1118 02:55:28.412468 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1118 02:55:28.412546 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1118 02:55:28.419973 (+ 7427us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1118 02:55:28.624911 (+204938us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35090: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":46974,"thread_start_us":55089,"threads_started":1}
W20261118 02:55:28.625101 3689 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:28.625598 5654 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:28.376008 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.2.229.66:40377)
1118 02:55:28.412964 (+ 36956us) server_negotiation.cc:207] Beginning negotiation
1118 02:55:28.412970 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1118 02:55:28.412989 (+ 19us) server_negotiation.cc:408] Connection header received
1118 02:55:28.413060 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1118 02:55:28.413066 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1118 02:55:28.413136 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1118 02:55:28.413253 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1118 02:55:28.420292 (+ 7039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1118 02:55:28.421747 (+ 1455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1118 02:55:28.625541 (+203794us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50116: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":36839,"thread_start_us":43627,"threads_started":1}
W20261118 02:55:28.625998 3955 connection.cc:573] client connection to 127.2.229.125:38339 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 02:55:28.626674 3955 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.229.125:38339)
W20261118 02:55:28.627084 3955 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.229.125:38339)
W20261118 02:55:28.627597 3955 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.229.125:38339)
W20261118 02:55:28.627985 3955 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.229.125:38339)
W20261118 02:55:28.628552 5631 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 02:55:28.626985 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.2.229.125:38339 (local address 127.2.229.67:33901)
1118 02:55:28.627137 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1118 02:55:28.627153 (+ 16us) client_negotiation.cc:175] Beginning negotiation
1118 02:55:28.627340 (+ 187us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 02:55:28.628508 (+ 1168us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":75}
I20261118 02:55:28.918025 5673 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:28.918331 5673 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:28.920075 5673 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:28.921134 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:28.922744 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:28.922981 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:28.923113 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:28.923457 5673 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:55:28.959977 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 1: this message will repeat every 5th retry.
W20261118 02:55:29.164395 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 1: this message will repeat every 5th retry.
I20261118 02:55:29.624663 5687 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:29.624853 5687 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:29.625605 5687 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:29.627727 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:29.627923 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:29.628039 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:29.628325 5687 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:55:30.372432 5701 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:30.372648 5701 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:30.373390 5701 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:30.375844 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:30.376075 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:30.376238 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:30.376500 5701 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:55:30.459797 2965 master-stress-test.cc:579] Killing test thread 1/25
W20261118 02:55:30.576287 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 6: this message will repeat every 5th retry.
W20261118 02:55:31.364048 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 6: this message will repeat every 5th retry.
I20261118 02:55:31.934599 5727 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:31.934803 5727 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:31.935712 5727 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:31.937803 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:31.938073 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:31.938205 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:31.938476 5727 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:55:32.381196 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 6: this message will repeat every 5th retry.
W20261118 02:55:32.388866 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 6: this message will repeat every 5th retry.
I20261118 02:55:33.085836 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:55:33.086050 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:33.448501 5750 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:33.448701 5750 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:33.449447 5750 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:33.451268 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:33.451478 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:33.451635 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:33.451882 5750 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:55:33.669234 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 6: this message will repeat every 5th retry.
I20261118 02:55:35.902992 5670 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:55:35.903170 5670 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:35.904042 5670 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:35.904940 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:55:35.905107 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:35.905292 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:35.905550 5670 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:36.518183 5784 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:36.518404 5784 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:36.519074 5784 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:36.519963 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20261118 02:55:36.521373 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:36.521543 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:36.521661 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:36.521893 5784 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:55:36.796990 5670 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:55:36.797199 5670 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:36.797950 5670 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:36.798799 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:55:36.798962 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:55:36.799101 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:36.799361 5670 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:38.087471 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:55:38.087716 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:55:39.108853 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 6: this message will repeat every 5th retry.
W20261118 02:55:39.387306 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 6: this message will repeat every 5th retry.
I20261118 02:55:40.308772 5833 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:40.308964 5833 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:40.309700 5833 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:40.311770 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:40.311973 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:40.312115 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:40.312369 5833 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:55:40.539942 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 11: this message will repeat every 5th retry.
W20261118 02:55:41.168836 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 11: this message will repeat every 5th retry.
W20261118 02:55:42.403877 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 11: this message will repeat every 5th retry.
W20261118 02:55:43.032873 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 11: this message will repeat every 5th retry.
I20261118 02:55:43.089282 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:55:43.089504 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:55:43.577225 5845 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:55:43.577451 5845 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:43.578172 5845 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:43.579113 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:55:43.579312 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:43.579429 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:43.579654 5845 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:55:44.005230 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 11: this message will repeat every 5th retry.
I20261118 02:55:45.397172 5885 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:45.397377 5885 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:45.398104 5885 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:45.399135 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 02:55:45.399145 5555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:45.400573 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:55:45.400753 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:55:45.400882 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:45.401109 5885 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:55:47.205823 5908 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:55:47.206045 5908 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:47.206779 5908 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:47.207985 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:55:47.208194 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:55:47.208325 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:47.208623 5908 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:48.091045 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:55:48.091236 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:55:48.731650 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 11: this message will repeat every 5th retry.
W20261118 02:55:49.752238 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 11: this message will repeat every 5th retry.
I20261118 02:55:50.034221 5932 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:55:50.034430 5932 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:50.035189 5932 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:50.036211 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:55:50.036546 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:50.036669 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:50.036922 5932 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:55:50.311463 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 16: this message will repeat every 5th retry.
W20261118 02:55:51.239516 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 16: this message will repeat every 5th retry.
W20261118 02:55:52.350610 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 16: this message will repeat every 5th retry.
W20261118 02:55:52.828927 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 16: this message will repeat every 5th retry.
I20261118 02:55:53.092849 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:55:53.093044 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:55:54.411811 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 16: this message will repeat every 5th retry.
I20261118 02:55:55.992439 5993 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:55:55.992622 5993 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:55:55.993381 5993 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:55:55.994601 5555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:55.996445 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:55:55.996683 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:55:55.996791 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:55:55.997062 5993 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:55:56.657282 5983 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:55:56.657473 5983 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:55:56.658321 5983 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:56.659269 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:55:56.659467 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:55:56.659564 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:56.659798 5983 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:57.104182 5983 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:55:57.104408 5983 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:55:57.105095 5983 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:55:57.106101 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:55:57.106492 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:55:57.106614 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:55:57.106855 5983 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:55:58.094657 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:55:58.094882 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:55:58.128293 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 16: this message will repeat every 5th retry.
W20261118 02:55:58.964799 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 16: this message will repeat every 5th retry.
W20261118 02:55:59.645067 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 21: this message will repeat every 5th retry.
W20261118 02:56:00.229477 3665 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (65 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:56:00.396555 3802 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (144 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:56:00.462844 3947 heartbeater.cc:648] Failed to heartbeat to 127.2.229.126:38025 (134 consecutive failures): Network error: Failed to ping master at 127.2.229.126:38025: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:56:00.605662 6045 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:56:00.605907 6045 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:56:00.606544 6045 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:56:00.607568 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:56:00.607733 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:56:00.607924 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:56:00.608206 6045 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:56:00.935271 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 21: this message will repeat every 5th retry.
W20261118 02:56:02.621012 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 21: this message will repeat every 5th retry.
W20261118 02:56:02.815258 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 21: this message will repeat every 5th retry.
I20261118 02:56:03.096472 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:03.096679 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:04.525568 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 21: this message will repeat every 5th retry.
I20261118 02:56:06.938094 6134 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:56:06.938328 6134 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:56:06.939091 6134 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:56:06.940845 5555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:56:06.942695 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:56:06.943400 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:56:06.943528 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:56:06.943857 6134 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:56:07.982607 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 21: this message will repeat every 5th retry.
I20261118 02:56:08.098405 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:08.098644 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:08.637887 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 21: this message will repeat every 5th retry.
W20261118 02:56:10.016618 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 26: this message will repeat every 5th retry.
W20261118 02:56:10.580835 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 26: this message will repeat every 5th retry.
W20261118 02:56:12.255952 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 26: this message will repeat every 5th retry.
I20261118 02:56:13.100533 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:13.100728 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:13.172420 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 26: this message will repeat every 5th retry.
W20261118 02:56:14.392789 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 26: this message will repeat every 5th retry.
I20261118 02:56:14.622538 6182 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:56:14.622720 6182 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:56:14.623581 6182 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:56:14.624326 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:56:14.624482 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:56:14.624579 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:56:14.624786 6182 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:56:16.350860 6236 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:56:16.351069 6236 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:56:16.351869 6236 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:56:16.352778 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:56:16.354365 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:56:16.354550 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:56:16.354700 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:56:16.354986 6236 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:56:18.102280 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:18.102491 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:18.447316 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 26: this message will repeat every 5th retry.
W20261118 02:56:18.752430 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 26: this message will repeat every 5th retry.
I20261118 02:56:19.769289 6182 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:56:19.769528 6182 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:56:19.770141 6182 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:56:19.771028 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:56:19.771209 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:56:19.771368 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:56:19.771590 6182 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:56:20.447530 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 31: this message will repeat every 5th retry.
W20261118 02:56:20.683594 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 31: this message will repeat every 5th retry.
W20261118 02:56:22.858696 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 31: this message will repeat every 5th retry.
I20261118 02:56:22.976730 6182 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:56:22.976922 6182 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:56:22.977869 6182 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:56:22.978744 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:56:22.978987 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:56:22.979118 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:56:22.979377 6182 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:56:23.104084 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:23.104324 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:56:23.131151 6182 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:56:23.131319 6182 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:56:23.131963 6182 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:56:23.132817 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:56:23.132997 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:56:23.133113 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:56:23.133387 6182 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:56:23.556182 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 31: this message will repeat every 5th retry.
W20261118 02:56:24.113736 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 31: this message will repeat every 5th retry.
W20261118 02:56:27.429958 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 31: this message will repeat every 5th retry.
I20261118 02:56:28.106338 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:28.106529 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:28.928059 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 31: this message will repeat every 5th retry.
I20261118 02:56:30.336418 6390 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:56:30.336602 6390 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:56:30.337323 6390 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:56:30.338326 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:56:30.339444 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:56:30.339612 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:56:30.339736 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:56:30.339988 6390 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:56:30.887456 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 36: this message will repeat every 5th retry.
W20261118 02:56:31.233016 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 36: this message will repeat every 5th retry.
W20261118 02:56:32.689002 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 36: this message will repeat every 5th retry.
W20261118 02:56:33.011636 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 36: this message will repeat every 5th retry.
I20261118 02:56:33.108178 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:33.108383 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:33.850236 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 36: this message will repeat every 5th retry.
W20261118 02:56:37.732273 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 36: this message will repeat every 5th retry.
I20261118 02:56:38.110023 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:38.110229 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:38.483652 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 36: this message will repeat every 5th retry.
W20261118 02:56:40.218343 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 41: this message will repeat every 5th retry.
W20261118 02:56:40.434991 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 41: this message will repeat every 5th retry.
I20261118 02:56:41.089998 6485 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:56:41.090183 6485 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:56:41.091089 6485 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:56:41.092043 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:56:41.092242 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:56:41.092366 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:56:41.092660 6485 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:56:42.392079 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 41: this message will repeat every 5th retry.
I20261118 02:56:42.624784 6506 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:56:42.625013 6506 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:56:42.625800 6506 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:56:42.626959 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:56:42.627652 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:56:42.627781 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:56:42.628055 6506 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:56:42.841060 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 41: this message will repeat every 5th retry.
I20261118 02:56:43.112290 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:43.112517 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:44.553699 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 41: this message will repeat every 5th retry.
I20261118 02:56:46.179849 6506 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:56:46.180191 6506 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:56:46.181200 6506 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:56:46.182098 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:56:46.182281 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:56:46.182389 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:56:46.182653 6506 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:56:48.114272 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:48.114487 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:48.187366 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 41: this message will repeat every 5th retry.
W20261118 02:56:48.359508 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 41: this message will repeat every 5th retry.
I20261118 02:56:49.149977 6592 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:56:49.150317 6592 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:56:49.151283 6592 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:56:49.152418 5555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:56:49.153699 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:56:49.153858 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:56:49.153991 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:56:49.154255 6592 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:56:50.503849 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 46: this message will repeat every 5th retry.
W20261118 02:56:50.921205 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 46: this message will repeat every 5th retry.
W20261118 02:56:51.940369 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 46: this message will repeat every 5th retry.
I20261118 02:56:53.115981 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:53.116219 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:53.304440 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 46: this message will repeat every 5th retry.
W20261118 02:56:54.531152 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 46: this message will repeat every 5th retry.
I20261118 02:56:56.163416 6506 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:56:56.163666 6506 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:56:56.164624 6506 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:56:56.165562 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:56:56.165759 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:56:56.165907 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:56:56.166239 6506 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:56:57.913360 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 46: this message will repeat every 5th retry.
I20261118 02:56:58.117758 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:56:58.117944 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:56:58.396592 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 46: this message will repeat every 5th retry.
I20261118 02:56:59.625134 6692 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:56:59.625365 6692 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:56:59.626210 6692 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:56:59.627149 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:56:59.628471 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:56:59.628677 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:56:59.628805 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:56:59.629104 6692 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:57:00.409845 6506 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:57:00.410043 6506 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:57:00.410719 6506 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:57:00.411444 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:57:00.411804 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:57:00.411923 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:57:00.412189 6506 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:57:00.416695 3665 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (185 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:57:00.675436 3947 heartbeater.cc:648] Failed to heartbeat to 127.2.229.126:38025 (361 consecutive failures): Network error: Failed to ping master at 127.2.229.126:38025: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:57:00.694525 3802 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (410 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:57:00.888291 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 51: this message will repeat every 5th retry.
W20261118 02:57:01.359127 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 51: this message will repeat every 5th retry.
W20261118 02:57:02.072229 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 51: this message will repeat every 5th retry.
I20261118 02:57:03.119546 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:03.119764 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:03.234062 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 51: this message will repeat every 5th retry.
W20261118 02:57:05.344251 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 51: this message will repeat every 5th retry.
I20261118 02:57:08.121307 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:08.121505 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:08.717660 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 51: this message will repeat every 5th retry.
W20261118 02:57:09.268417 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 51: this message will repeat every 5th retry.
I20261118 02:57:09.348333 6794 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:57:09.348572 6794 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:57:09.349795 6794 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:57:09.350687 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:57:09.351788 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:57:09.351953 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:57:09.352061 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:57:09.352324 6794 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:57:10.313709 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 56: this message will repeat every 5th retry.
W20261118 02:57:11.008773 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 56: this message will repeat every 5th retry.
W20261118 02:57:12.083596 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 56: this message will repeat every 5th retry.
W20261118 02:57:12.877405 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 56: this message will repeat every 5th retry.
I20261118 02:57:13.122869 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:13.123042 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:57:13.527782 6844 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:57:13.527964 6844 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:57:13.528836 6844 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:57:13.529757 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:57:13.529932 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:57:13.530035 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:57:13.530251 6844 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:57:13.782737 6844 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:57:13.782931 6844 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:57:13.783641 6844 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:57:13.784574 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:57:13.784770 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:57:13.784873 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:57:13.785085 6844 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:57:15.527200 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 56: this message will repeat every 5th retry.
I20261118 02:57:18.124677 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:18.124908 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:18.888922 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 56: this message will repeat every 5th retry.
W20261118 02:57:19.224823 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 61: this message will repeat every 5th retry.
W20261118 02:57:19.643155 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 56: this message will repeat every 5th retry.
I20261118 02:57:21.382807 6935 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:57:21.383019 6935 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:57:21.383836 6935 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:57:21.385071 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:57:21.386317 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:57:21.386483 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:57:21.386593 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:57:21.386862 6935 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:57:21.638518 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 61: this message will repeat every 5th retry.
I20261118 02:57:22.537374 6844 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:57:22.537598 6844 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:57:22.538316 6844 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:57:22.539288 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:57:22.539484 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:57:22.539587 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:57:22.539827 6844 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:57:22.608337 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 61: this message will repeat every 5th retry.
I20261118 02:57:23.059811 6958 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:57:23.060045 6958 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:57:23.060866 6958 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:57:23.062950 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:57:23.063333 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:57:23.063444 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:57:23.063720 6958 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:57:23.126088 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:23.126300 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:23.227061 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 61: this message will repeat every 5th retry.
W20261118 02:57:25.723588 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 61: this message will repeat every 5th retry.
I20261118 02:57:27.518970 6996 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:57:27.519127 6996 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:57:27.519719 6996 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:57:27.520565 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:57:27.520710 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:57:27.520828 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:57:27.521050 6996 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:57:28.086007 5559 maintenance_manager.cc:419] P 813a6937cea442ef81645fccc32207f4: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033343
I20261118 02:57:28.127781 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:28.127975 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:57:28.133642 5558 maintenance_manager.cc:643] P 813a6937cea442ef81645fccc32207f4: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.047s user 0.025s sys 0.007s Metrics: {"bytes_written":27552,"cfile_init":1,"dirs.queue_time_us":395,"dirs.run_cpu_time_us":326,"dirs.run_wall_time_us":1120,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":74,"thread_start_us":853,"threads_started":2,"wal-append.queue_time_us":680}
I20261118 02:57:28.134949 5559 maintenance_manager.cc:419] P 813a6937cea442ef81645fccc32207f4: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4582 bytes on disk
I20261118 02:57:28.135690 5558 maintenance_manager.cc:643] P 813a6937cea442ef81645fccc32207f4: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20261118 02:57:28.257326 6996 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:57:28.257539 6996 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:57:28.258230 6996 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:57:28.259096 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:57:28.259222 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:57:28.259339 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:57:28.259589 6996 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:57:29.153463 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 61: this message will repeat every 5th retry.
W20261118 02:57:29.218022 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 66: this message will repeat every 5th retry.
W20261118 02:57:29.278539 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 61: this message will repeat every 5th retry.
W20261118 02:57:32.317335 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 66: this message will repeat every 5th retry.
W20261118 02:57:33.106753 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 66: this message will repeat every 5th retry.
I20261118 02:57:33.129535 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:33.129786 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:33.193670 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 66: this message will repeat every 5th retry.
W20261118 02:57:35.607445 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 66: this message will repeat every 5th retry.
I20261118 02:57:38.131350 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:38.131604 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:38.915306 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 71: this message will repeat every 5th retry.
W20261118 02:57:39.266597 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 66: this message will repeat every 5th retry.
W20261118 02:57:39.697613 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 66: this message will repeat every 5th retry.
I20261118 02:57:40.594195 7202 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:57:40.594416 7202 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:57:40.595391 7202 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:57:40.596397 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 02:57:40.598003 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:57:40.598182 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:57:40.598346 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:57:40.598616 7202 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:57:42.236785 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 71: this message will repeat every 5th retry.
W20261118 02:57:42.904325 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 71: this message will repeat every 5th retry.
I20261118 02:57:43.133165 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:43.133348 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:43.382508 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 71: this message will repeat every 5th retry.
I20261118 02:57:44.654345 7105 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:57:44.654580 7105 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:57:44.655544 7105 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:57:44.656658 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:57:44.656867 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:57:44.656953 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:57:44.657202 7105 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:57:45.286657 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 71: this message will repeat every 5th retry.
I20261118 02:57:47.943230 7105 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:57:47.943408 7105 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:57:47.944044 7105 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:57:47.945016 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:57:47.945175 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:57:47.945299 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:57:47.945525 7304 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:57:48.135010 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:48.135201 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:48.875918 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 76: this message will repeat every 5th retry.
W20261118 02:57:49.223453 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 71: this message will repeat every 5th retry.
W20261118 02:57:49.766619 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 71: this message will repeat every 5th retry.
I20261118 02:57:52.349977 7360 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:57:52.350210 7360 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:57:52.350996 7360 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:57:52.352134 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:57:52.353381 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:57:52.353543 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:57:52.353725 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:57:52.354013 7360 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:57:52.404729 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 76: this message will repeat every 5th retry.
I20261118 02:57:53.136974 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:53.137172 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:53.254227 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 76: this message will repeat every 5th retry.
W20261118 02:57:53.552588 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 76: this message will repeat every 5th retry.
W20261118 02:57:54.932454 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 76: this message will repeat every 5th retry.
I20261118 02:57:58.138635 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:57:58.138833 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:57:59.018529 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 81: this message will repeat every 5th retry.
I20261118 02:57:59.559317 7419 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:57:59.559527 7419 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:57:59.560314 7419 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:57:59.561225 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:57:59.561384 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:57:59.561528 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:57:59.561766 7419 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:57:59.565672 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 76: this message will repeat every 5th retry.
W20261118 02:58:00.144567 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 76: this message will repeat every 5th retry.
W20261118 02:58:00.616329 3665 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (305 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:58:00.958791 3949 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (571 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:58:00.973905 3802 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (683 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:58:02.822342 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 81: this message will repeat every 5th retry.
I20261118 02:58:02.847697 7482 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:58:02.847905 7482 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:58:02.848784 7482 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:02.849704 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:58:02.849859 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:58:02.849999 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:02.850291 7482 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:58:02.888734 7482 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:58:02.888943 7482 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:58:02.889647 7482 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:02.890311 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:58:02.890470 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:58:02.890573 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:02.890769 7482 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:58:03.050968 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 81: this message will repeat every 5th retry.
I20261118 02:58:03.140501 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:03.140720 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:03.245304 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 81: this message will repeat every 5th retry.
I20261118 02:58:05.032606 7516 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:58:05.032799 7516 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:58:05.033535 7516 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:58:05.034482 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:58:05.035732 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:58:05.035881 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:58:05.036039 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:58:05.036317 7516 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:58:05.130908 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 81: this message will repeat every 5th retry.
I20261118 02:58:06.861006 7545 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:58:06.861192 7545 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:58:06.861899 7545 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:06.862658 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:58:06.862838 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:58:06.862974 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:06.863242 7545 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:58:08.142208 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:08.142498 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:08.980360 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 81: this message will repeat every 5th retry.
W20261118 02:58:08.997622 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 86: this message will repeat every 5th retry.
W20261118 02:58:10.549921 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 81: this message will repeat every 5th retry.
W20261118 02:58:12.724419 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 86: this message will repeat every 5th retry.
I20261118 02:58:13.144183 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:13.144383 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:13.223852 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 86: this message will repeat every 5th retry.
W20261118 02:58:13.358853 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 86: this message will repeat every 5th retry.
I20261118 02:58:13.973117 7545 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:58:13.973318 7545 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:58:13.973928 7545 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:13.974880 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:58:13.975062 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:58:13.975190 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:13.975451 7545 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:58:15.452023 7660 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:58:15.452253 7660 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:58:15.453040 7660 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:58:15.454002 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:58:15.455380 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:58:15.455614 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:58:15.455732 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:58:15.456010 7660 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:58:15.732462 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 86: this message will repeat every 5th retry.
I20261118 02:58:18.146245 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:18.146437 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:18.375077 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 91: this message will repeat every 5th retry.
W20261118 02:58:19.783526 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 86: this message will repeat every 5th retry.
W20261118 02:58:20.572118 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 86: this message will repeat every 5th retry.
W20261118 02:58:22.920607 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 91: this message will repeat every 5th retry.
W20261118 02:58:23.047988 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 91: this message will repeat every 5th retry.
I20261118 02:58:23.148067 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:23.148340 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:23.243335 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 91: this message will repeat every 5th retry.
I20261118 02:58:23.716390 7776 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:58:23.716588 7776 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:58:23.717418 7776 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:58:23.718448 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:58:23.719712 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:58:23.719890 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:58:23.719997 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:58:23.720254 7776 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:58:24.423086 7715 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:58:24.423274 7715 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:58:24.424043 7715 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:24.424883 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:58:24.425033 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:58:24.425149 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:24.425343 7715 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:58:25.530254 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 91: this message will repeat every 5th retry.
I20261118 02:58:28.150002 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:28.150254 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:28.207217 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 96: this message will repeat every 5th retry.
I20261118 02:58:28.532006 7715 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:58:28.532238 7715 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:58:28.532825 7715 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:28.533812 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:58:28.533962 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:58:28.534099 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:28.534385 7715 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:58:29.613914 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 91: this message will repeat every 5th retry.
W20261118 02:58:30.316879 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 91: this message will repeat every 5th retry.
W20261118 02:58:32.397710 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 96: this message will repeat every 5th retry.
W20261118 02:58:32.811247 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 96: this message will repeat every 5th retry.
I20261118 02:58:33.151935 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:33.152170 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:33.206007 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 96: this message will repeat every 5th retry.
W20261118 02:58:35.088420 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 96: this message will repeat every 5th retry.
I20261118 02:58:35.644779 7917 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:58:35.644985 7917 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:58:35.645798 7917 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:58:35.646842 5555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:58:35.648344 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:58:35.648530 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:58:35.648663 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:58:35.648937 7917 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:58:36.542936 7851 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:58:36.543102 7851 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:58:36.543743 7851 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:36.544659 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:58:36.544800 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:58:36.544909 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:36.545130 7851 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:58:38.025601 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 101: this message will repeat every 5th retry.
I20261118 02:58:38.153779 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:38.154428 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:58:39.628973 7943 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:58:39.629137 7943 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:58:39.630405 7943 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:39.631377 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:58:39.631531 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:58:39.631620 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:39.631844 7943 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:58:39.912817 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 96: this message will repeat every 5th retry.
W20261118 02:58:40.110134 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 96: this message will repeat every 5th retry.
I20261118 02:58:40.765530 7990 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:58:40.765715 7990 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:58:40.766443 7990 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:58:40.767242 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:58:40.768546 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:58:40.768729 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:58:40.768850 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:58:40.769096 7990 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:58:42.661867 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 101: this message will repeat every 5th retry.
I20261118 02:58:43.156122 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:43.156363 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:43.337124 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 101: this message will repeat every 5th retry.
W20261118 02:58:43.376363 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 101: this message will repeat every 5th retry.
I20261118 02:58:43.912972 7943 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:58:43.913167 7943 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:58:43.913892 7943 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:43.914738 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:58:43.914909 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:58:43.915009 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:43.915249 7943 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:58:44.596998 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 101: this message will repeat every 5th retry.
W20261118 02:58:47.813879 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 106: this message will repeat every 5th retry.
I20261118 02:58:48.119235 8080 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:58:48.119484 8080 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:58:48.120294 8080 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:58:48.121352 5555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:58:48.122648 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:58:48.122849 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:58:48.122973 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:58:48.123236 8080 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:58:48.157598 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:48.157836 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:49.309739 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 101: this message will repeat every 5th retry.
W20261118 02:58:49.761521 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 101: this message will repeat every 5th retry.
I20261118 02:58:50.941085 7943 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:58:50.941250 7943 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:58:50.941870 7943 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:50.942737 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:58:50.942862 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:58:50.942963 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:50.943217 7943 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:58:52.901240 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 106: this message will repeat every 5th retry.
I20261118 02:58:53.159374 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:53.159590 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:53.184293 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 106: this message will repeat every 5th retry.
W20261118 02:58:53.250200 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 106: this message will repeat every 5th retry.
W20261118 02:58:54.610394 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 106: this message will repeat every 5th retry.
I20261118 02:58:55.165832 8157 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:58:55.166010 8157 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:58:55.166731 8157 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:58:55.167569 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:58:55.168864 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:58:55.169199 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:58:55.169319 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:58:55.169595 8157 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:58:58.157732 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 111: this message will repeat every 5th retry.
I20261118 02:58:58.160801 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:58:58.160995 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:58:59.485185 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 106: this message will repeat every 5th retry.
I20261118 02:58:59.602751 8182 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:58:59.603106 8182 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:58:59.604077 8182 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:58:59.604846 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:58:59.604982 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:58:59.605069 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:58:59.605238 8182 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:59:00.474730 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 106: this message will repeat every 5th retry.
W20261118 02:59:00.793666 3665 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (425 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:59:01.046530 3949 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (794 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:59:01.386315 3802 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (951 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:59:02.635401 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 111: this message will repeat every 5th retry.
W20261118 02:59:02.814745 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 111: this message will repeat every 5th retry.
I20261118 02:59:03.162468 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:03.162700 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 02:59:03.315689 8182 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:59:03.315883 8182 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:59:03.316619 8182 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:03.317687 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:59:03.318092 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:59:03.318188 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:03.318475 8182 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:59:04.083133 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 111: this message will repeat every 5th retry.
W20261118 02:59:04.494549 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 111: this message will repeat every 5th retry.
I20261118 02:59:04.678925 8253 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:59:04.679127 8253 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:59:04.679788 8253 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:04.680799 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:59:04.681092 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:59:04.681190 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:04.681406 8253 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:59:06.879763 8277 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:59:06.879956 8277 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:59:06.880702 8277 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:59:06.881628 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 02:59:06.882915 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:59:06.883103 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:59:06.883216 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:59:06.883445 8277 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:59:07.705319 8253 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:59:07.705505 8253 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:59:07.706146 8253 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:07.707072 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:59:07.707245 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:59:07.707355 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:07.707620 8253 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:59:07.984573 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 116: this message will repeat every 5th retry.
I20261118 02:59:08.164357 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:08.164602 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:09.783567 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 111: this message will repeat every 5th retry.
W20261118 02:59:10.100283 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 111: this message will repeat every 5th retry.
I20261118 02:59:10.226858 8316 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:59:10.227104 8316 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:59:10.227859 8316 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:59:10.230258 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:59:10.230464 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:59:10.230607 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:59:10.230877 8316 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:59:12.440300 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 116: this message will repeat every 5th retry.
W20261118 02:59:12.704334 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 116: this message will repeat every 5th retry.
I20261118 02:59:12.794190 8350 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:59:12.794389 8350 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:59:12.795086 8350 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:59:12.796036 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 02:59:12.797224 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:59:12.797408 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:59:12.797526 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:59:12.797791 8350 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:59:13.165732 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:13.165930 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:13.383304 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 116: this message will repeat every 5th retry.
W20261118 02:59:13.880877 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 116: this message will repeat every 5th retry.
I20261118 02:59:14.775440 8373 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:59:14.775702 8373 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:59:14.776453 8373 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:59:14.778760 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:59:14.778945 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:59:14.779065 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:59:14.779287 8373 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:59:17.219662 8253 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:59:17.219847 8253 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:59:17.220664 8253 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:17.221486 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:59:17.221627 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:59:17.221725 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:17.221954 8253 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:59:18.167467 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:18.167695 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:18.357481 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 121: this message will repeat every 5th retry.
I20261118 02:59:18.581887 8253 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:59:18.582090 8253 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:59:18.582803 8253 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:18.583911 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:59:18.584656 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:59:18.584745 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:18.584959 8253 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:59:20.007988 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 116: this message will repeat every 5th retry.
W20261118 02:59:20.645378 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 116: this message will repeat every 5th retry.
W20261118 02:59:22.600922 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 121: this message will repeat every 5th retry.
W20261118 02:59:23.137544 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 121: this message will repeat every 5th retry.
I20261118 02:59:23.169693 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:23.169919 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:23.474047 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 121: this message will repeat every 5th retry.
W20261118 02:59:24.151412 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 121: this message will repeat every 5th retry.
I20261118 02:59:27.873066 8496 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:59:27.873217 8496 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:59:27.873821 8496 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:27.874632 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:59:27.874763 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:59:27.874897 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:27.875106 8496 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:59:28.171466 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:28.171648 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:28.240993 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 126: this message will repeat every 5th retry.
W20261118 02:59:29.834745 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 121: this message will repeat every 5th retry.
W20261118 02:59:30.827389 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 121: this message will repeat every 5th retry.
I20261118 02:59:31.229068 8496 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:59:31.229236 8496 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:59:31.229826 8496 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:31.230625 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:59:31.230741 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:59:31.230862 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:31.231081 8496 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:59:32.790480 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 126: this message will repeat every 5th retry.
I20261118 02:59:32.855837 8561 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:59:32.856045 8561 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:59:32.856999 8561 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:59:32.858637 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 02:59:32.859872 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:59:32.860028 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:59:32.860193 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:59:32.860458 8561 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 02:59:32.952739 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 126: this message will repeat every 5th retry.
I20261118 02:59:33.172973 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:33.173195 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:33.334769 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 126: this message will repeat every 5th retry.
W20261118 02:59:33.904570 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 126: this message will repeat every 5th retry.
I20261118 02:59:35.417480 8582 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:59:35.417649 8582 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:59:35.418402 8582 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:59:35.421079 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:59:35.421217 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:59:35.421314 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:59:35.421484 8582 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:59:36.381064 8567 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:59:36.381253 8567 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:59:36.382048 8567 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:36.382725 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:59:36.382855 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:59:36.382943 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:36.383144 8567 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 02:59:38.174733 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:38.174988 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:38.235090 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 131: this message will repeat every 5th retry.
W20261118 02:59:39.591650 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 126: this message will repeat every 5th retry.
W20261118 02:59:40.303997 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 126: this message will repeat every 5th retry.
I20261118 02:59:41.319796 8567 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:59:41.320008 8567 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:59:41.320714 8567 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:41.321551 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:59:41.321691 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:59:41.321800 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:41.322021 8567 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:59:43.006160 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 131: this message will repeat every 5th retry.
W20261118 02:59:43.125178 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 131: this message will repeat every 5th retry.
W20261118 02:59:43.166941 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 131: this message will repeat every 5th retry.
I20261118 02:59:43.176337 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:43.176496 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:43.211259 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 131: this message will repeat every 5th retry.
I20261118 02:59:43.725240 8663 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:59:43.725435 8663 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:59:43.726222 8663 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:59:43.727092 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 02:59:43.728869 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:59:43.729112 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:59:43.729187 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:59:43.729442 8663 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:59:47.651849 8708 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:59:47.652037 8708 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:59:47.652654 8708 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:59:47.654553 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 02:59:47.654690 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 02:59:47.654790 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:59:47.655011 8708 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:59:48.177984 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:48.178169 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:48.391852 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 136: this message will repeat every 5th retry.
W20261118 02:59:49.471623 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 131: this message will repeat every 5th retry.
W20261118 02:59:50.582154 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 131: this message will repeat every 5th retry.
I20261118 02:59:51.978672 8728 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 02:59:51.978927 8728 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 02:59:51.979714 8728 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:51.980540 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 02:59:51.980695 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 02:59:51.980810 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:51.980988 8728 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:59:52.198585 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 136: this message will repeat every 5th retry.
I20261118 02:59:52.921782 8728 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 02:59:52.921998 8728 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 02:59:52.922627 8728 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 02:59:52.923377 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 02:59:52.923636 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 02:59:52.923750 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 02:59:52.923982 8728 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 02:59:53.053478 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 136: this message will repeat every 5th retry.
I20261118 02:59:53.179688 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:53.179870 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:53.523923 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 136: this message will repeat every 5th retry.
W20261118 02:59:53.575775 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 136: this message will repeat every 5th retry.
I20261118 02:59:57.373075 8794 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 02:59:57.373271 8794 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 02:59:57.374028 8794 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 02:59:57.375049 5555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 02:59:57.376375 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 02:59:57.376556 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 02:59:57.376715 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 02:59:57.376955 8794 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 02:59:58.181721 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 02:59:58.181943 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 02:59:58.394097 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 141: this message will repeat every 5th retry.
W20261118 02:59:59.626570 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 136: this message will repeat every 5th retry.
W20261118 03:00:00.265889 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 136: this message will repeat every 5th retry.
W20261118 03:00:00.933529 3665 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (545 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 03:00:01.445329 3949 heartbeater.cc:648] Failed to heartbeat to 127.2.229.125:38339 (1022 consecutive failures): Network error: Failed to ping master at 127.2.229.125:38339: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 03:00:01.534240 3800 heartbeater.cc:648] Failed to heartbeat to 127.2.229.126:38025 (1244 consecutive failures): Network error: Failed to ping master at 127.2.229.126:38025: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 03:00:01.534797 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 141: this message will repeat every 5th retry.
I20261118 03:00:03.183622 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 03:00:03.183862 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 03:00:03.427086 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 141: this message will repeat every 5th retry.
I20261118 03:00:03.739813 8869 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 03:00:03.740005 8869 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 03:00:03.740705 8869 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 03:00:03.741703 5555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 03:00:03.743103 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
W20261118 03:00:03.743252 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
I20261118 03:00:03.743361 5555 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 03:00:03.743626 8869 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 03:00:03.811874 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 141: this message will repeat every 5th retry.
W20261118 03:00:03.989079 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 141: this message will repeat every 5th retry.
I20261118 03:00:04.976514 8833 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 03:00:04.976691 8833 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 03:00:04.977308 8833 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 03:00:04.978266 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 03:00:04.978451 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 03:00:04.978546 3552 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 03:00:04.978761 8833 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 03:00:07.900988 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 146: this message will repeat every 5th retry.
I20261118 03:00:08.185303 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 03:00:08.185462 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 03:00:10.116611 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 141: this message will repeat every 5th retry.
I20261118 03:00:10.539739 8910 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 03:00:10.539944 8910 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 03:00:10.540849 8910 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 03:00:10.541875 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 03:00:10.542029 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 03:00:10.542129 3553 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 03:00:10.542333 8910 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 03:00:10.730866 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 141: this message will repeat every 5th retry.
W20261118 03:00:11.152266 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 146: this message will repeat every 5th retry.
I20261118 03:00:11.578753 8938 raft_consensus.cc:499] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99a39b1d9d814deeb3a8a644e9b392ef)
I20261118 03:00:11.578953 8938 raft_consensus.cc:521] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } } peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } }
I20261118 03:00:11.579612 8938 leader_election.cc:290] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 03:00:11.580538 3552 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
W20261118 03:00:11.585196 3553 leader_election.cc:343] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: a013d180f2cd430ab1ba30b39fd27591
I20261118 03:00:11.585372 3553 leader_election.cc:304] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 03:00:11.585675 8938 raft_consensus.cc:2771] T a013d180f2cd430ab1ba30b39fd27591 P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 03:00:13.009191 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 146: this message will repeat every 5th retry.
I20261118 03:00:13.187103 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 03:00:13.187258 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 03:00:13.862955 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 146: this message will repeat every 5th retry.
W20261118 03:00:14.415006 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 146: this message will repeat every 5th retry.
W20261118 03:00:17.568670 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 151: this message will repeat every 5th retry.
I20261118 03:00:18.188782 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 03:00:18.188958 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 03:00:18.405079 8938 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 03:00:18.405267 8938 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 03:00:18.405918 8938 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 03:00:18.406755 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 03:00:18.406901 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 03:00:18.407008 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 03:00:18.407233 8938 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 03:00:20.692251 3838 consensus_peers.cc:597] T 2b01d7e596bd4790b8ba5dc88d8e12cc P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b01d7e596bd4790b8ba5dc88d8e12cc. This is attempt 146: this message will repeat every 5th retry.
W20261118 03:00:20.773861 3691 consensus_peers.cc:597] T 8ae679d2feb04b95ab2f4fb51e443d64 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae679d2feb04b95ab2f4fb51e443d64. This is attempt 151: this message will repeat every 5th retry.
W20261118 03:00:20.833590 3691 consensus_peers.cc:597] T 0e901a03b90e4c978eb98be87163cfe5 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e901a03b90e4c978eb98be87163cfe5. This is attempt 146: this message will repeat every 5th retry.
I20261118 03:00:22.821983 9055 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 03:00:22.822163 9055 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 03:00:22.823007 9055 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 03:00:22.824005 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 03:00:22.825198 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 03:00:22.825367 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 03:00:22.825480 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 03:00:22.825721 9055 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20261118 03:00:23.190071 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 03:00:23.190238 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 03:00:23.469233 3691 consensus_peers.cc:597] T ffe8d4c111d14054b76f9c7724072a94 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe8d4c111d14054b76f9c7724072a94. This is attempt 151: this message will repeat every 5th retry.
W20261118 03:00:24.134795 3838 consensus_peers.cc:597] T 973831392ec24e0681c3d4fffd8bb199 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 973831392ec24e0681c3d4fffd8bb199. This is attempt 151: this message will repeat every 5th retry.
W20261118 03:00:24.519721 3838 consensus_peers.cc:597] T c0713dcf79f54873a99d7556ad8f1454 P 3f1973986ec1466eab5bec01dd81654f -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0713dcf79f54873a99d7556ad8f1454. This is attempt 151: this message will repeat every 5th retry.
I20261118 03:00:24.894520 8938 raft_consensus.cc:499] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f1973986ec1466eab5bec01dd81654f)
I20261118 03:00:24.894719 8938 raft_consensus.cc:521] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fdf7d7ece324fefb3a0c376add5b7e8" member_type: VOTER last_known_addr { host: "127.2.229.65" port: 33705 } } peers { permanent_uuid: "99a39b1d9d814deeb3a8a644e9b392ef" member_type: VOTER last_known_addr { host: "127.2.229.66" port: 40377 } } peers { permanent_uuid: "3f1973986ec1466eab5bec01dd81654f" member_type: VOTER last_known_addr { host: "127.2.229.67" port: 42237 } }
I20261118 03:00:24.895377 8938 leader_election.cc:290] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377), 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237)
W20261118 03:00:24.896324 3553 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f1973986ec1466eab5bec01dd81654f (127.2.229.67:42237): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
W20261118 03:00:24.896512 3552 leader_election.cc:343] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99a39b1d9d814deeb3a8a644e9b392ef (127.2.229.66:40377): Not found: Tablet not found: c92c29c30ab6411dabe6f02331d29b7a
I20261118 03:00:24.896611 3552 leader_election.cc:304] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fdf7d7ece324fefb3a0c376add5b7e8; no voters: 3f1973986ec1466eab5bec01dd81654f, 99a39b1d9d814deeb3a8a644e9b392ef
I20261118 03:00:24.896838 8938 raft_consensus.cc:2771] T c92c29c30ab6411dabe6f02331d29b7a P 1fdf7d7ece324fefb3a0c376add5b7e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 03:00:28.191579 5526 master_runner.cc:187] Error getting master registration for 127.2.229.126:38025: Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111),
I20261118 03:00:28.191725 5526 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 03:00:28.381434 3691 consensus_peers.cc:597] T b82ca8a282f94392bb2746ed7b8b8ba7 P 99a39b1d9d814deeb3a8a644e9b392ef -> Peer 1fdf7d7ece324fefb3a0c376add5b7e8 (127.2.229.65:33705): Couldn't send request to peer 1fdf7d7ece324fefb3a0c376add5b7e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82ca8a282f94392bb2746ed7b8b8ba7. This is attempt 156: this message will repeat every 5th retry.
I20261118 03:00:28.420482 5559 maintenance_manager.cc:419] P 813a6937cea442ef81645fccc32207f4: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4582 bytes on disk
I20261118 03:00:28.425880 5558 maintenance_manager.cc:643] P 813a6937cea442ef81645fccc32207f4: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":332,"dirs.run_cpu_time_us":89,"dirs.run_wall_time_us":90,"thread_start_us":480,"threads_started":2,"wal-append.queue_time_us":333}
I20261118 03:00:28.504838 9149 raft_consensus.cc:499] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 457eb38cba564ecd9f6f11245172260b)
I20261118 03:00:28.505012 9149 raft_consensus.cc:521] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6b20fa203404a88463ea27bdd3c7d" member_type: VOTER last_known_addr { host: "127.2.229.126" port: 38025 } } peers { permanent_uuid: "457eb38cba564ecd9f6f11245172260b" member_type: VOTER last_known_addr { host: "127.2.229.125" port: 38339 } } peers { permanent_uuid: "813a6937cea442ef81645fccc32207f4" member_type: VOTER last_known_addr { host: "127.2.229.124" port: 36695 } }
I20261118 03:00:28.505633 9149 leader_election.cc:290] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025), 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339)
W20261118 03:00:28.506227 5555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 03:00:28.507370 5555 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 457eb38cba564ecd9f6f11245172260b (127.2.229.125:38339): Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 03:00:28.507586 5556 leader_election.cc:336] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26c6b20fa203404a88463ea27bdd3c7d (127.2.229.126:38025): Network error: Client connection negotiation failed: client connection to 127.2.229.126:38025: connect: Connection refused (error 111)
I20261118 03:00:28.507695 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 813a6937cea442ef81645fccc32207f4; no voters: 26c6b20fa203404a88463ea27bdd3c7d, 457eb38cba564ecd9f6f11245172260b
I20261118 03:00:28.507944 9149 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 813a6937cea442ef81645fccc32207f4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20261118 03:00:28.633126 3956 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 03:00:28.633379 3956 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 03:00:28.633582 3956 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
W20261118 03:00:28.633705 3956 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
F20261118 03:00:28.644670 3968 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_52fca7a2e65f4e679bbef9de3df569f1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Aborted at 1794970828 (unix time) try "date -d @1794970828" if you are using GNU date ***
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: T*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650697 3971 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ee3815d9524243ec886bd171ccaff9e2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650697 3971 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ee3815d9524243ec886bd171ccaff9e2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650763 3988 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000b95) received by PID 2965 (TID 0x7f8796674700) from PID 2965; stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650697 3971 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ee3815d9524243ec886bd171ccaff9e2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650763 3988 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.651935 3978 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f87ac1f4980 (unknown) at ??:0
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650697 3971 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ee3815d9524243ec886bd171ccaff9e2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650763 3988 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.651935 3978 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653137 3976 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653162 3973 master-stress-test.cc:330] Check failed: _s.ok() Bad status: T*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650697 3971 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ee3815d9524243ec886bd171ccaff9e2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650763 3988 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.651935 3978 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653137 3976 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653162 3973 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650697 3971 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ee3815d9524243ec886bd171ccaff9e2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650763 3988 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.651935 3978 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653137 3976 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653162 3973 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653373 3981 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650697 3971 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ee3815d9524243ec886bd171ccaff9e2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650763 3988 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.651935 3978 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653137 3976 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653162 3973 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653373 3981 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653424 3967 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e39f59e2fdbb47fda4c10beba29ad8f1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650697 3971 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ee3815d9524243ec886bd171ccaff9e2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650763 3988 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.651935 3978 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653137 3976 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653162 3973 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653373 3981 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653424 3967 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e39f59e2fdbb47fda4c10beba29ad8f1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653592 3974 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20261118 03:00:28.644738 3979 master-stress-test.cc:353] Check failed: _s.ok() Bad status: F20261118 03:00:28.644739 Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.644862 3972 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646270 3964 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_51d36ff687ca47b5995e5693d7b8931b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646327 3982 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.646515 3966 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bdbed231860946c2bf9b514f724bc3e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648559 3975 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648626 3986 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.648727 3987 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.649066 3962 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dc982811b980416484416db1f74c685d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650036 3977 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650137 3969 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1d3bb660e14d4101996960be8caa8f81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650185 3965 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fd4fcac1f1d946fa898176f1d5e87e9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3983 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650679 3963 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8139578347f041afbe51671d3be78447 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650697 3971 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ee3815d9524243ec886bd171ccaff9e2 on the master: CreateTable timed out after deadline expired: CreateTable passed its @ 0x7f87a4f64e87 gsignal
deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.650763 3988 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.651935 3978 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653137 3976 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653162 3973 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653373 3981 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653424 3967 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e39f59e2fdbb47fda4c10beba29ad8f1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653592 3974 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)F20261118 03:00:28.653714 3970 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_40868189cecf45d0b705461c4328ee17 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.229.126:38025,127.2.229.125:38339,127.2.229.124:36695, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.2.229.125:38339: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f87a4f667f1 abort at ??:0
@ 0x7f87a7cb3d5d google::LogMessage::Fail() at ??:0
@ 0x7f87a7cb7b23 google::LogMessage::SendToLog() at ??:0
@ 0x7f87a7cb375c google::LogMessage::Flush() at ??:0
@ 0x7f87a7cb4ee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x409062 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:299
@ 0x447484 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44733d std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x4472fa _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x4472ae std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x446fe9 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f87a55ee6df (unknown) at ??:0
@ 0x7f87ac1e96db start_thread at ??:0
@ 0x7f87a504761f clone at ??:0