Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:02:08.044346 10114 test_util.cc:268] Using random seed: -291355377
2024-05-01T14:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:02:08Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskmQb6Mr/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskmQb6Mr/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/security.properties
2024-05-01 14:02:14: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskmQb6Mr/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskmQb6Mr/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20240501 14:02:22.861047 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.190:34475
--webserver_interface=127.9.224.190
--webserver_port=0
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 14:02:23.195191 10357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:23.195528 10357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:23.195650 10357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:23.203382 10357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:23.203516 10357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:23.203562 10357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:23.203644 10357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:23.213593 10357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.190:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.224.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:23.215027 10357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:23.216951 10357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:23.230497 10365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:23.230482 10362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:23.231122 10363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:23.231585 10357 server_base.cc:1034] running on GCE node
I20240501 14:02:23.232481 10357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:23.234222 10357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:23.235467 10357 hybrid_clock.cc:648] HybridClock initialized: now 1714572143235405 us; error 40 us; skew 500 ppm
I20240501 14:02:23.235848 10357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:23.238552 10357 webserver.cc:460] Webserver started at http://127.9.224.190:35165/ using document root <none> and password file <none>
I20240501 14:02:23.239336 10357 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:23.239456 10357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:23.239764 10357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:23.243961 10357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/instance:
uuid: "e47f54722b654655bfc871851b93abbc"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-gtmz"
I20240501 14:02:23.244812 10357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal/instance:
uuid: "e47f54722b654655bfc871851b93abbc"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-gtmz"
I20240501 14:02:23.250746 10357 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20240501 14:02:23.254951 10371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:23.256781 10357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240501 14:02:23.257042 10357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
uuid: "e47f54722b654655bfc871851b93abbc"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-gtmz"
I20240501 14:02:23.257247 10357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:23.275274 10357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:23.276217 10357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:23.276559 10357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:23.295064 10357 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.190:34475
I20240501 14:02:23.295095 10422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.190:34475 every 8 connection(s)
I20240501 14:02:23.296654 10357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
I20240501 14:02:23.298643 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 10357
I20240501 14:02:23.298918 10114 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal/instance
I20240501 14:02:23.300647 10423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.301606 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.189:42871
--webserver_interface=127.9.224.189
--webserver_port=0
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240501 14:02:23.307253 10423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:02:23.326035 10423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:02:23.327414 10372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:23.329118 10423 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.189:42871: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:23.378515 10423 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } attempt: 1
W20240501 14:02:23.380626 10423 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.189:42871: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:23.448915 10423 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } attempt: 2
W20240501 14:02:23.451117 10423 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.189:42871: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:23.542400 10423 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } attempt: 3
W20240501 14:02:23.544376 10423 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.189:42871: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:23.643410 10424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:23.643699 10424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:23.643831 10424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:23.651851 10424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:23.651966 10424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:23.652020 10424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:23.652096 10424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:23.661391 10424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.189:42871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.224.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:23.662794 10424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:23.664520 10424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:23.675993 10432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:23.676048 10435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:23.676523 10433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:23.677002 10424 server_base.cc:1034] running on GCE node
I20240501 14:02:23.677747 10424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:23.679191 10424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:23.680457 10424 hybrid_clock.cc:648] HybridClock initialized: now 1714572143680351 us; error 49 us; skew 500 ppm
I20240501 14:02:23.680805 10424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:23.683216 10424 webserver.cc:460] Webserver started at http://127.9.224.189:38899/ using document root <none> and password file <none>
I20240501 14:02:23.683954 10424 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:23.684078 10424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:23.684374 10424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:23.687084 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/data/instance:
uuid: "e048fbcc4e544adea37b6bd9e5ac8543"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-gtmz"
I20240501 14:02:23.687604 10423 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } attempt: 4
I20240501 14:02:23.687891 10424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/wal/instance:
uuid: "e048fbcc4e544adea37b6bd9e5ac8543"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-gtmz"
W20240501 14:02:23.689652 10423 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.189:42871: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:23.693775 10424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:02:23.697460 10441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:23.699357 10424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20240501 14:02:23.699582 10424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/wal
uuid: "e048fbcc4e544adea37b6bd9e5ac8543"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-gtmz"
I20240501 14:02:23.699877 10424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:23.731546 10424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:23.732501 10424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:23.732815 10424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:23.754097 10424 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.189:42871
I20240501 14:02:23.754146 10492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.189:42871 every 8 connection(s)
I20240501 14:02:23.755584 10424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/data/info.pb
I20240501 14:02:23.760561 10493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.763909 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 10424
I20240501 14:02:23.764087 10114 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-1/wal/instance
I20240501 14:02:23.765246 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--webserver_interface=127.9.224.188
--webserver_port=0
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240501 14:02:23.768412 10493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:02:23.786981 10493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:02:23.798552 10493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:02:23.799805 10445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
W20240501 14:02:23.801481 10493 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.188:38257: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
I20240501 14:02:23.850796 10493 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } attempt: 1
W20240501 14:02:23.853153 10493 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.188:38257: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
I20240501 14:02:23.921456 10493 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } attempt: 2
W20240501 14:02:23.923377 10493 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.188:38257: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
I20240501 14:02:23.988941 10423 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } attempt: 5
I20240501 14:02:24.000938 10423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } has no permanent_uuid. Determining permanent_uuid...
W20240501 14:02:24.002822 10423 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.188:38257: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
I20240501 14:02:24.014636 10493 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } attempt: 3
W20240501 14:02:24.017134 10493 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.188:38257: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
I20240501 14:02:24.054153 10423 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } attempt: 1
W20240501 14:02:24.056488 10423 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.188:38257: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
W20240501 14:02:24.110455 10494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:24.110785 10494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:24.110909 10494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:24.118928 10494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:24.119076 10494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:24.119153 10494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:24.119235 10494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:24.124763 10423 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } attempt: 2
W20240501 14:02:24.126653 10423 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.188:38257: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
I20240501 14:02:24.128723 10494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.224.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:24.130311 10494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:24.132117 10494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:24.142808 10502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:24.143273 10505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:24.143507 10503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:24.143807 10494 server_base.cc:1034] running on GCE node
I20240501 14:02:24.144662 10494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:24.146174 10494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:24.147434 10494 hybrid_clock.cc:648] HybridClock initialized: now 1714572144147352 us; error 56 us; skew 500 ppm
I20240501 14:02:24.147795 10494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:24.150183 10494 webserver.cc:460] Webserver started at http://127.9.224.188:35625/ using document root <none> and password file <none>
I20240501 14:02:24.150918 10494 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:24.151055 10494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:24.151348 10494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:24.154242 10494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/instance:
uuid: "4163d103537745edb461e4dbdda724e6"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-gtmz"
I20240501 14:02:24.155011 10494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal/instance:
uuid: "4163d103537745edb461e4dbdda724e6"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-gtmz"
I20240501 14:02:24.160358 10493 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } attempt: 4
W20240501 14:02:24.162134 10493 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.9.224.188:38257: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
I20240501 14:02:24.162537 10494 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240501 14:02:24.167245 10511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:24.169579 10494 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:24.169883 10494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
uuid: "4163d103537745edb461e4dbdda724e6"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-gtmz"
I20240501 14:02:24.170121 10494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:24.202795 10494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:24.203667 10494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:24.203956 10494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:24.224170 10494 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.188:38257
I20240501 14:02:24.224195 10562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.188:38257 every 8 connection(s)
I20240501 14:02:24.225721 10494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
I20240501 14:02:24.229802 10563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:24.232913 10423 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } attempt: 3
I20240501 14:02:24.235790 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 10494
I20240501 14:02:24.236025 10114 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal/instance
I20240501 14:02:24.236758 10563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:02:24.250871 10563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:02:24.255753 10423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap starting.
I20240501 14:02:24.258826 10563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } has no permanent_uuid. Determining permanent_uuid...
I20240501 14:02:24.261301 10423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:24.263017 10423 log.cc:824] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:24.269670 10423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: No bootstrap required, opened a new log
I20240501 14:02:24.271965 10563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap starting.
I20240501 14:02:24.277662 10563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:24.279332 10563 log.cc:824] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:24.279256 10423 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:24.279708 10423 raft_consensus.cc:374] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:24.279861 10423 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47f54722b654655bfc871851b93abbc, State: Initialized, Role: FOLLOWER
I20240501 14:02:24.280575 10423 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:24.281705 10572 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.282018 10572 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:24.282721 10563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: No bootstrap required, opened a new log
I20240501 14:02:24.283044 10423 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:24.294366 10563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:24.294930 10563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:24.295097 10563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4163d103537745edb461e4dbdda724e6, State: Initialized, Role: FOLLOWER
I20240501 14:02:24.296097 10563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:24.301223 10586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.301609 10586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:24.303619 10563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:24.316316 10586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:24.316717 10586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:24.318691 10586 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
I20240501 14:02:24.320489 10398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4163d103537745edb461e4dbdda724e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47f54722b654655bfc871851b93abbc" is_pre_election: true
I20240501 14:02:24.321718 10398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4163d103537745edb461e4dbdda724e6 in term 0.
I20240501 14:02:24.322641 10513 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters:
I20240501 14:02:24.323287 10586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:24.323444 10586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:24.323551 10586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:24.326923 10586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:24.329133 10398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4163d103537745edb461e4dbdda724e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47f54722b654655bfc871851b93abbc"
I20240501 14:02:24.329404 10398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:24.333249 10586 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 1 election: Requested vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
I20240501 14:02:24.332957 10398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4163d103537745edb461e4dbdda724e6 in term 1.
I20240501 14:02:24.334478 10513 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters:
I20240501 14:02:24.335078 10586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:24.338560 10586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 1 LEADER]: Becoming Leader. State: Replica: 4163d103537745edb461e4dbdda724e6, State: Running, Role: LEADER
I20240501 14:02:24.339748 10586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
W20240501 14:02:24.346942 10587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 14:02:24.347105 10587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 14:02:24.314067 10357 master_runner.cc:186] Error getting master registration for 127.9.224.189:42871: OK, instance_id { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" instance_seqno: 1714572143748412 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 14:02:24.358282 10357 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:24.383896 10592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4163d103537745edb461e4dbdda724e6. Latest consensus state: current_term: 1 leader_uuid: "4163d103537745edb461e4dbdda724e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.384330 10592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:24.402352 10596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:24.430431 10596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240501 14:02:24.456413 10517 tablet.cc:2369] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:24.461330 10493 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } attempt: 5
I20240501 14:02:24.480834 10493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: Bootstrap starting.
I20240501 14:02:24.499022 10493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:24.503998 10493 log.cc:824] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:24.506325 10398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 1 FOLLOWER]: Refusing update from remote peer 4163d103537745edb461e4dbdda724e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:24.507465 10586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:24.525661 10493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: No bootstrap required, opened a new log
W20240501 14:02:24.530048 10377 tablet.cc:2369] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:24.536957 10493 raft_consensus.cc:348] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:24.537482 10493 raft_consensus.cc:374] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:24.537627 10493 raft_consensus.cc:729] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e048fbcc4e544adea37b6bd9e5ac8543, State: Initialized, Role: FOLLOWER
I20240501 14:02:24.538697 10493 consensus_queue.cc:260] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:24.539075 10572 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4163d103537745edb461e4dbdda724e6. Latest consensus state: current_term: 1 leader_uuid: "4163d103537745edb461e4dbdda724e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.539383 10572 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:24.545828 10608 sys_catalog.cc:455] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.546198 10608 sys_catalog.cc:458] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:24.548116 10493 sys_catalog.cc:564] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:24.548185 10468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4163d103537745edb461e4dbdda724e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" is_pre_election: true
I20240501 14:02:24.548629 10466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:24.548138 10467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4163d103537745edb461e4dbdda724e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e048fbcc4e544adea37b6bd9e5ac8543"
I20240501 14:02:24.552479 10600 mvcc.cc:205] Tried to move back new op lower bound from 7022887503878676480 to 7022887503228350464. Current Snapshot: MvccSnapshot[applied={T|T < 7022887503878676480}]
I20240501 14:02:24.552551 10466 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 1 FOLLOWER]: Refusing update from remote peer 4163d103537745edb461e4dbdda724e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:02:24.552920 10586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4163d103537745edb461e4dbdda724e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.553582 10586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:24.553515 10592 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:24.555743 10572 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4163d103537745edb461e4dbdda724e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.556026 10572 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:24.564289 10596 catalog_manager.cc:1347] Generated new cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:24.564440 10596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:24.614940 10608 sys_catalog.cc:455] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4163d103537745edb461e4dbdda724e6. Latest consensus state: current_term: 1 leader_uuid: "4163d103537745edb461e4dbdda724e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.615293 10608 sys_catalog.cc:458] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:24.617107 10617 sys_catalog.cc:455] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4163d103537745edb461e4dbdda724e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.617558 10617 sys_catalog.cc:458] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:24.624519 10586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4163d103537745edb461e4dbdda724e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:24.624883 10586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:24.636543 10596 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:02:24.638033 10596 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:24.654426 10596 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Generated new TSK 0
I20240501 14:02:24.655788 10596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:24.704368 10494 master_runner.cc:186] Error getting master registration for 127.9.224.189:42871: OK, instance_id { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" instance_seqno: 1714572143748412 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 14:02:24.714563 10494 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:24.725230 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.224.129:0
--local_ip_for_outbound_sockets=127.9.224.129
--webserver_interface=127.9.224.129
--webserver_port=0
--tserver_master_addrs=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240501 14:02:24.765959 10636 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:24.766098 10636 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: loading cluster ID for follower catalog manager: success
I20240501 14:02:24.770519 10636 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: acquiring CA information for follower catalog manager: success
I20240501 14:02:24.782967 10636 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240501 14:02:25.200631 10637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:25.200958 10637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:25.201134 10637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:25.209365 10637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:25.209591 10637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.224.129
I20240501 14:02:25.219677 10637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.224.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.224.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.224.129
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:25.221335 10637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:25.223615 10637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:25.241498 10643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.242408 10644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.253990 10646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:25.255219 10637 server_base.cc:1034] running on GCE node
I20240501 14:02:25.256346 10637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:25.257902 10637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:25.259209 10637 hybrid_clock.cc:648] HybridClock initialized: now 1714572145259119 us; error 71 us; skew 500 ppm
I20240501 14:02:25.259622 10637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:25.262737 10637 webserver.cc:460] Webserver started at http://127.9.224.129:42909/ using document root <none> and password file <none>
I20240501 14:02:25.263765 10637 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:25.263923 10637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:25.264292 10637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:25.268090 10637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/data/instance:
uuid: "5e09841056a14c29be2546cc1d0b4643"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-gtmz"
I20240501 14:02:25.269169 10637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal/instance:
uuid: "5e09841056a14c29be2546cc1d0b4643"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-gtmz"
I20240501 14:02:25.276232 10637 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20240501 14:02:25.280961 10652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.282708 10637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:25.282912 10637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal
uuid: "5e09841056a14c29be2546cc1d0b4643"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-gtmz"
I20240501 14:02:25.283140 10637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:25.305636 10637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:25.306779 10637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:25.307227 10637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:25.308419 10637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:25.310468 10637 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:25.310570 10637 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.310688 10637 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:25.310741 10637 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.344285 10637 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.129:38539
I20240501 14:02:25.344321 10764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.129:38539 every 8 connection(s)
I20240501 14:02:25.345647 10637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/data/info.pb
I20240501 14:02:25.349573 10587 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:25.349694 10587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: loading cluster ID for follower catalog manager: success
I20240501 14:02:25.353569 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 10637
I20240501 14:02:25.353811 10114 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal/instance
I20240501 14:02:25.357036 10587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: acquiring CA information for follower catalog manager: success
I20240501 14:02:25.361474 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.224.130:0
--local_ip_for_outbound_sockets=127.9.224.130
--webserver_interface=127.9.224.130
--webserver_port=0
--tserver_master_addrs=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240501 14:02:25.367833 10587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:25.411027 10766 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:25.411178 10765 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:25.411257 10767 heartbeater.cc:347] Connected to a master server at 127.9.224.189:42871
I20240501 14:02:25.411389 10765 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:25.411393 10766 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:25.411422 10767 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:25.412606 10767 heartbeater.cc:511] Master 127.9.224.189:42871 requested a full tablet report, sending...
I20240501 14:02:25.412671 10766 heartbeater.cc:511] Master 127.9.224.190:34475 requested a full tablet report, sending...
I20240501 14:02:25.412642 10765 heartbeater.cc:511] Master 127.9.224.188:38257 requested a full tablet report, sending...
I20240501 14:02:25.414950 10528 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:25.415091 10458 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:25.415105 10388 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:25.416785 10528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.224.129:45855
I20240501 14:02:25.559445 10776 raft_consensus.cc:482] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4163d103537745edb461e4dbdda724e6)
I20240501 14:02:25.559705 10776 raft_consensus.cc:504] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:25.561378 10776 leader_election.cc:288] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
I20240501 14:02:25.561975 10398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e47f54722b654655bfc871851b93abbc" is_pre_election: true
I20240501 14:02:25.562322 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6" is_pre_election: true
I20240501 14:02:25.563099 10445 leader_election.cc:302] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e048fbcc4e544adea37b6bd9e5ac8543; no voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc
I20240501 14:02:25.563422 10776 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:02:25.712280 10775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:25.712684 10775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:25.712855 10775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:25.723675 10775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:25.723924 10775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.224.130
I20240501 14:02:25.735518 10775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.224.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.224.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.224.130
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:25.737387 10775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:25.739825 10775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:25.754863 10782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.755422 10781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:25.755999 10775 server_base.cc:1034] running on GCE node
W20240501 14:02:25.757297 10784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:25.758069 10775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:25.759086 10775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:25.760319 10775 hybrid_clock.cc:648] HybridClock initialized: now 1714572145760245 us; error 51 us; skew 500 ppm
I20240501 14:02:25.760615 10775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:25.763098 10775 webserver.cc:460] Webserver started at http://127.9.224.130:40713/ using document root <none> and password file <none>
I20240501 14:02:25.763803 10775 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:25.763924 10775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:25.764207 10775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:25.766873 10775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/data/instance:
uuid: "156c3e56589d4969b853b3b512301a48"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-gtmz"
I20240501 14:02:25.767585 10775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal/instance:
uuid: "156c3e56589d4969b853b3b512301a48"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-gtmz"
I20240501 14:02:25.775071 10775 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20240501 14:02:25.778950 10790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.780417 10775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:02:25.780587 10775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal
uuid: "156c3e56589d4969b853b3b512301a48"
format_stamp: "Formatted at 2024-05-01 14:02:25 on dist-test-slave-gtmz"
I20240501 14:02:25.780769 10775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:25.792678 10775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:25.793512 10775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:25.793800 10775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:25.794700 10775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:25.796239 10775 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:25.796315 10775 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.796432 10775 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:25.796483 10775 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.826364 10775 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.130:38403
I20240501 14:02:25.826400 10902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.130:38403 every 8 connection(s)
I20240501 14:02:25.827593 10775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/data/info.pb
I20240501 14:02:25.834929 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 10775
I20240501 14:02:25.835156 10114 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal/instance
I20240501 14:02:25.839249 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.224.131:0
--local_ip_for_outbound_sockets=127.9.224.131
--webserver_interface=127.9.224.131
--webserver_port=0
--tserver_master_addrs=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240501 14:02:25.845974 10903 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:25.845932 10904 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:25.846326 10905 heartbeater.cc:347] Connected to a master server at 127.9.224.189:42871
I20240501 14:02:25.846350 10903 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:25.846352 10904 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:25.846583 10905 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:25.847734 10904 heartbeater.cc:511] Master 127.9.224.190:34475 requested a full tablet report, sending...
I20240501 14:02:25.847785 10905 heartbeater.cc:511] Master 127.9.224.189:42871 requested a full tablet report, sending...
I20240501 14:02:25.847831 10903 heartbeater.cc:511] Master 127.9.224.188:38257 requested a full tablet report, sending...
I20240501 14:02:25.849452 10458 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:25.849498 10388 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:25.850741 10528 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:25.851534 10528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.224.130:41617
I20240501 14:02:25.919518 10765 heartbeater.cc:503] Master 127.9.224.188:38257 was elected leader, sending a full tablet report...
W20240501 14:02:26.162386 10910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:26.162727 10910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:26.162866 10910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:26.170348 10910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:26.170558 10910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.224.131
I20240501 14:02:26.179646 10910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.224.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.224.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.224.131
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:26.180954 10910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:26.182651 10910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:26.194801 10915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:26.195129 10916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:26.195391 10918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:26.195866 10910 server_base.cc:1034] running on GCE node
I20240501 14:02:26.196525 10910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:26.197578 10910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:26.198843 10910 hybrid_clock.cc:648] HybridClock initialized: now 1714572146198767 us; error 47 us; skew 500 ppm
I20240501 14:02:26.199139 10910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:26.201588 10910 webserver.cc:460] Webserver started at http://127.9.224.131:35111/ using document root <none> and password file <none>
I20240501 14:02:26.202313 10910 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:26.202430 10910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:26.202679 10910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:26.205504 10910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/data/instance:
uuid: "c1d9952ad6a24cecbcde3d02a59c9a75"
format_stamp: "Formatted at 2024-05-01 14:02:26 on dist-test-slave-gtmz"
I20240501 14:02:26.206293 10910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal/instance:
uuid: "c1d9952ad6a24cecbcde3d02a59c9a75"
format_stamp: "Formatted at 2024-05-01 14:02:26 on dist-test-slave-gtmz"
I20240501 14:02:26.211627 10910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:26.215269 10924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:26.216743 10910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:26.216957 10910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal
uuid: "c1d9952ad6a24cecbcde3d02a59c9a75"
format_stamp: "Formatted at 2024-05-01 14:02:26 on dist-test-slave-gtmz"
I20240501 14:02:26.217239 10910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:26.235052 10910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:26.235935 10910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:26.236239 10910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:26.237269 10910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:26.238828 10910 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:26.238906 10910 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:26.239037 10910 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:26.239089 10910 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:26.269491 10910 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.131:35371
I20240501 14:02:26.269528 11036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.131:35371 every 8 connection(s)
I20240501 14:02:26.270699 10910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/data/info.pb
I20240501 14:02:26.279177 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 10910
I20240501 14:02:26.279383 10114 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal/instance
I20240501 14:02:26.288151 11037 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:26.288300 11038 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:26.288146 11039 heartbeater.cc:347] Connected to a master server at 127.9.224.189:42871
I20240501 14:02:26.288532 11037 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:26.288612 11039 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:26.288527 11038 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:26.289778 11037 heartbeater.cc:511] Master 127.9.224.188:38257 requested a full tablet report, sending...
I20240501 14:02:26.289808 11039 heartbeater.cc:511] Master 127.9.224.189:42871 requested a full tablet report, sending...
I20240501 14:02:26.289837 11038 heartbeater.cc:511] Master 127.9.224.190:34475 requested a full tablet report, sending...
I20240501 14:02:26.291209 10528 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:26.291294 10388 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:26.291978 10528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.224.131:50575
I20240501 14:02:26.292727 10458 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:26.305958 10114 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 14:02:26.339795 10526 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_d3a10d5d2bfb4854bd055ff3e0768f4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.339825 10525 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_381d9eb21d47411f8927a03d0f96a650"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.340823 10527 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_d67cd2442dc24ffc8d4dc86e19a50315"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.341413 10528 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_cc9ce1d1031f42789771ae5eb4a6df43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.341852 10524 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_e4a61f4822d448919e6decf48ba63a8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.344305 10522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_740f0f0aef9c46919f5658f6d2be21e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.344323 10523 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_6d8f3ef2579a49df82d7ae9d6fbb3e2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.346441 10521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_498fa9129a934033b9b363449b003bef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.346824 10520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_c9596eb80e944f919a8c85174a7080ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 14:02:26.353468 10525 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_381d9eb21d47411f8927a03d0f96a650 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:02:26.353688 10903 heartbeater.cc:503] Master 127.9.224.188:38257 was elected leader, sending a full tablet report...
I20240501 14:02:26.357820 10519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52604:
name: "default.Table_30033735ecf54950a6eab1ae05ef1e55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.377890 10114 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu pid 10494
I20240501 14:02:26.379361 10558 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:26.495106 11084 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4163d103537745edb461e4dbdda724e6)
I20240501 14:02:26.502722 11084 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:26.505766 10466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" is_pre_election: true
I20240501 14:02:26.506935 11084 leader_election.cc:288] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
I20240501 14:02:26.514323 10538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6" is_pre_election: true
I20240501 14:02:26.515039 10375 leader_election.cc:302] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e47f54722b654655bfc871851b93abbc; no voters: 4163d103537745edb461e4dbdda724e6, e048fbcc4e544adea37b6bd9e5ac8543
I20240501 14:02:26.515516 10114 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu with pid 10494
I20240501 14:02:26.516222 11084 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:26.533421 11085 raft_consensus.cc:482] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4163d103537745edb461e4dbdda724e6)
I20240501 14:02:26.533739 11085 raft_consensus.cc:504] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:26.534619 11085 leader_election.cc:288] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
I20240501 14:02:26.535095 10398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e47f54722b654655bfc871851b93abbc" is_pre_election: true
I20240501 14:02:26.535357 10398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e048fbcc4e544adea37b6bd9e5ac8543 in term 1.
I20240501 14:02:26.535862 10443 leader_election.cc:302] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc; no voters:
I20240501 14:02:26.536231 11085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:02:26.536341 11085 raft_consensus.cc:482] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4163d103537745edb461e4dbdda724e6)
I20240501 14:02:26.536427 11085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:26.543654 11085 raft_consensus.cc:504] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:26.544911 10398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e47f54722b654655bfc871851b93abbc"
I20240501 14:02:26.545192 10398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:26.547806 10398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e048fbcc4e544adea37b6bd9e5ac8543 in term 2.
I20240501 14:02:26.548426 10443 leader_election.cc:302] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc; no voters:
I20240501 14:02:26.554875 11085 leader_election.cc:288] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [CANDIDATE]: Term 2 election: Requested vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
I20240501 14:02:26.555651 11085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:26.555809 11085 raft_consensus.cc:686] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [term 2 LEADER]: Becoming Leader. State: Replica: e048fbcc4e544adea37b6bd9e5ac8543, State: Running, Role: LEADER
I20240501 14:02:26.556350 11085 consensus_queue.cc:237] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
W20240501 14:02:26.568940 11047 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.224.188:38257)
W20240501 14:02:26.573172 10765 heartbeater.cc:644] Failed to heartbeat to 127.9.224.188:38257 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.9.224.188:38257 (error 108)
I20240501 14:02:26.573597 11086 sys_catalog.cc:455] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e048fbcc4e544adea37b6bd9e5ac8543. Latest consensus state: current_term: 2 leader_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:26.574018 11086 sys_catalog.cc:458] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: This master's current role is: LEADER
W20240501 14:02:26.574977 10445 connection.cc:535] client connection to 127.9.224.188:38257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:26.582276 11088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20240501 14:02:26.583978 11048 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:26.570527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.224.188:38257 (local address 127.0.0.1:52610)
0501 14:02:26.571153 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:26.571176 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:26.571346 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:26.582268 (+ 10922us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":356}
I20240501 14:02:26.584275 11088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:26.585286 11088 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:26.585355 11088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:26.586637 11088 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:26.587517 11088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: Loaded TSK: 0
I20240501 14:02:26.592571 11088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:26.593855 11088 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
W20240501 14:02:26.594596 11090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:26.581317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.224.188:38257 (local address 127.0.0.1:52624)
0501 14:02:26.592308 (+ 10991us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:26.592495 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":10868,"thread_start_us":11384,"threads_started":1}
I20240501 14:02:26.595021 11088 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
W20240501 14:02:26.595016 10445 leader_election.cc:334] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257): Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection reset by peer (error 104)
W20240501 14:02:26.595481 10445 leader_election.cc:334] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257): Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection reset by peer (error 104)
I20240501 14:02:26.605135 10457 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_c9596eb80e944f919a8c85174a7080ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.603140 10456 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_498fa9129a934033b9b363449b003bef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.604660 10450 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_e4a61f4822d448919e6decf48ba63a8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.603112 10458 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_30033735ecf54950a6eab1ae05ef1e55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.604454 10451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_381d9eb21d47411f8927a03d0f96a650"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.603276 10454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_d67cd2442dc24ffc8d4dc86e19a50315"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.603925 10452 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_d3a10d5d2bfb4854bd055ff3e0768f4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.603824 10453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_cc9ce1d1031f42789771ae5eb4a6df43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.604743 10449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_6d8f3ef2579a49df82d7ae9d6fbb3e2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:26.604841 10455 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_740f0f0aef9c46919f5658f6d2be21e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 14:02:26.619904 10456 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_498fa9129a934033b9b363449b003bef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 14:02:26.622927 10457 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c9596eb80e944f919a8c85174a7080ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 14:02:26.622980 10449 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 14:02:26.624018 10450 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e4a61f4822d448919e6decf48ba63a8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 14:02:26.624078 10453 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cc9ce1d1031f42789771ae5eb4a6df43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 14:02:26.625053 10452 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d3a10d5d2bfb4854bd055ff3e0768f4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 14:02:26.625283 10458 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_30033735ecf54950a6eab1ae05ef1e55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 14:02:26.626075 10454 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d67cd2442dc24ffc8d4dc86e19a50315 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 14:02:26.627033 10455 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_740f0f0aef9c46919f5658f6d2be21e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 14:02:26.627983 10451 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_381d9eb21d47411f8927a03d0f96a650 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:02:26.794131 11037 heartbeater.cc:503] Master 127.9.224.188:38257 was elected leader, sending a full tablet report...
W20240501 14:02:26.805977 11037 heartbeater.cc:644] Failed to heartbeat to 127.9.224.188:38257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
I20240501 14:02:26.843087 10456 catalog_manager.cc:2145] created HMS catalog entry for table [id=7c5ba258d7764086a1e0168310f43043]
I20240501 14:02:26.847551 10398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 2 FOLLOWER]: Refusing update from remote peer e048fbcc4e544adea37b6bd9e5ac8543: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:26.848340 11085 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:26.850598 10445 consensus_peers.cc:485] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 -> Peer 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257): Couldn't send request to peer 4163d103537745edb461e4dbdda724e6. Status: Network error: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:26.863123 11084 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e048fbcc4e544adea37b6bd9e5ac8543. Latest consensus state: current_term: 2 leader_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:26.863425 11084 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: FOLLOWER
W20240501 14:02:26.873435 10903 heartbeater.cc:644] Failed to heartbeat to 127.9.224.188:38257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.224.188:38257: connect: Connection refused (error 111)
I20240501 14:02:26.877202 11092 mvcc.cc:205] Tried to move back new op lower bound from 7022887513478172672 to 7022887512299618304. Current Snapshot: MvccSnapshot[applied={T|T < 7022887513478172672}]
I20240501 14:02:26.877998 11085 sys_catalog.cc:455] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:26.878700 11085 sys_catalog.cc:458] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543 [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:26.880851 10457 catalog_manager.cc:2145] created HMS catalog entry for table [id=8e7509fb5fb74cc1a065864a5721040f]
I20240501 14:02:26.886749 10456 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59138:
table { table_name: "default.Table_498fa9129a934033b9b363449b003bef" } new_table_name: "default.Table_ca677c5ce49e45d8ad3221682525d4c5" modify_external_catalogs: true
I20240501 14:02:26.891302 11084 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:26.891631 11084 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:26.925724 10449 catalog_manager.cc:2145] created HMS catalog entry for table [id=7115aaf817d14cd9ab186e9e9d789ca3]
I20240501 14:02:26.951823 10449 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59138:
table { table_name: "default.Table_c9596eb80e944f919a8c85174a7080ca" } new_table_name: "default.Table_fa62350979764cedbfc470287d68ffe1" modify_external_catalogs: true
I20240501 14:02:26.954488 10457 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59138:
table { table_name: "default.Table_6d8f3ef2579a49df82d7ae9d6fbb3e2b" } new_table_name: "default.Table_6cc7bd2d05ec418ba2a930d28e0257b0" modify_external_catalogs: true
I20240501 14:02:26.962199 10450 catalog_manager.cc:2145] created HMS catalog entry for table [id=3eeae6bd61554f2f8624397bf2db91cb]
I20240501 14:02:26.965564 10836 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddf192a1272427d899909c6bad20024 (DEFAULT_TABLE table=default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:26.967211 10836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ddf192a1272427d899909c6bad20024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:26.969453 10837 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8f988937a24d3a9dd3ff7d4ec8296d (DEFAULT_TABLE table=default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:26.969969 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8f988937a24d3a9dd3ff7d4ec8296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:26.964826 10838 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe711480111456e967d9f41986b56bb (DEFAULT_TABLE table=default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:26.977742 10838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe711480111456e967d9f41986b56bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:26.992549 10450 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59138:
table { table_name: "default.Table_e4a61f4822d448919e6decf48ba63a8d" } new_table_name: "default.Table_3bc74075f55d4af7a1e48e16115ae551" modify_external_catalogs: true
I20240501 14:02:26.993966 11106 tablet_bootstrap.cc:492] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:26.998682 10971 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8f988937a24d3a9dd3ff7d4ec8296d (DEFAULT_TABLE table=default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.000114 10971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8f988937a24d3a9dd3ff7d4ec8296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.000241 10838 tablet_service.cc:1467] Processing CreateTablet for tablet bb38cc63d5714e2b96371278ddc8e804 (DEFAULT_TABLE table=default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.001303 10970 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddf192a1272427d899909c6bad20024 (DEFAULT_TABLE table=default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.001508 10838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb38cc63d5714e2b96371278ddc8e804. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.001628 10970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ddf192a1272427d899909c6bad20024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.002274 11106 tablet_bootstrap.cc:654] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.003691 11106 log.cc:824] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:27.006603 10969 tablet_service.cc:1467] Processing CreateTablet for tablet bb38cc63d5714e2b96371278ddc8e804 (DEFAULT_TABLE table=default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.007102 10969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb38cc63d5714e2b96371278ddc8e804. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:26.997783 10972 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe711480111456e967d9f41986b56bb (DEFAULT_TABLE table=default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.012336 11106 tablet_bootstrap.cc:492] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.012678 10972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe711480111456e967d9f41986b56bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.012683 11106 ts_tablet_manager.cc:1367] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20240501 14:02:27.016686 10700 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe711480111456e967d9f41986b56bb (DEFAULT_TABLE table=default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.016458 10697 tablet_service.cc:1467] Processing CreateTablet for tablet bb38cc63d5714e2b96371278ddc8e804 (DEFAULT_TABLE table=default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.018117 10700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe711480111456e967d9f41986b56bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.018819 10699 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8f988937a24d3a9dd3ff7d4ec8296d (DEFAULT_TABLE table=default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.019306 10699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8f988937a24d3a9dd3ff7d4ec8296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.016710 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddf192a1272427d899909c6bad20024 (DEFAULT_TABLE table=default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.020223 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ddf192a1272427d899909c6bad20024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.024967 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb38cc63d5714e2b96371278ddc8e804. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.028867 11106 raft_consensus.cc:348] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.029438 11106 raft_consensus.cc:374] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.029597 11106 raft_consensus.cc:729] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.030422 11106 consensus_queue.cc:260] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.037912 11106 ts_tablet_manager.cc:1398] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.025s user 0.007s sys 0.007s
I20240501 14:02:27.039013 10905 heartbeater.cc:503] Master 127.9.224.189:42871 was elected leader, sending a full tablet report...
I20240501 14:02:27.041923 11110 tablet_bootstrap.cc:492] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.046208 11110 tablet_bootstrap.cc:654] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.047179 11109 tablet_bootstrap.cc:492] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.049538 11106 tablet_bootstrap.cc:492] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.051141 11106 tablet_bootstrap.cc:654] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.054754 11106 tablet_bootstrap.cc:492] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.054924 11106 ts_tablet_manager.cc:1367] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 14:02:27.055918 11106 raft_consensus.cc:348] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.056144 11106 raft_consensus.cc:374] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.056267 11106 raft_consensus.cc:729] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.056466 11109 tablet_bootstrap.cc:654] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.056531 11106 consensus_queue.cc:260] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.057169 11106 ts_tablet_manager.cc:1398] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:02:27.057466 11106 tablet_bootstrap.cc:492] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.059024 11106 tablet_bootstrap.cc:654] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.061991 11110 log.cc:824] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:27.063622 11109 log.cc:824] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:27.066685 11110 tablet_bootstrap.cc:492] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.066869 11109 tablet_bootstrap.cc:492] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.067135 11110 ts_tablet_manager.cc:1367] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.005s
I20240501 14:02:27.067178 11109 ts_tablet_manager.cc:1367] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240501 14:02:27.069001 11106 tablet_bootstrap.cc:492] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.069188 11106 ts_tablet_manager.cc:1367] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 14:02:27.069945 11106 raft_consensus.cc:348] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.070188 11106 raft_consensus.cc:374] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.070264 11106 raft_consensus.cc:729] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.070575 11106 consensus_queue.cc:260] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.074455 11106 ts_tablet_manager.cc:1398] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:02:27.076299 11110 raft_consensus.cc:348] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.076843 11110 raft_consensus.cc:374] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.077119 11110 raft_consensus.cc:729] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.077870 11106 tablet_bootstrap.cc:492] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.078058 11110 consensus_queue.cc:260] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.079998 11106 tablet_bootstrap.cc:654] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.080628 10453 catalog_manager.cc:2145] created HMS catalog entry for table [id=0ecb2a95ea54458d87a8e0361bd772b7]
W20240501 14:02:27.081681 10906 tablet.cc:2369] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:27.086048 11109 raft_consensus.cc:348] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.086555 11109 raft_consensus.cc:374] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.086678 11109 raft_consensus.cc:729] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.087428 11109 consensus_queue.cc:260] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.094154 10767 heartbeater.cc:503] Master 127.9.224.189:42871 was elected leader, sending a full tablet report...
I20240501 14:02:27.096371 11106 tablet_bootstrap.cc:492] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.096623 11106 ts_tablet_manager.cc:1367] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20240501 14:02:27.097503 11106 raft_consensus.cc:348] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.097803 11106 raft_consensus.cc:374] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.097944 11106 raft_consensus.cc:729] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.098299 11106 consensus_queue.cc:260] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.098955 11106 ts_tablet_manager.cc:1398] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:27.101882 10971 tablet_service.cc:1467] Processing CreateTablet for tablet 0496a42cfb744b25ad294d426ded937f (DEFAULT_TABLE table=default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.102084 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 0496a42cfb744b25ad294d426ded937f (DEFAULT_TABLE table=default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.102362 10971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0496a42cfb744b25ad294d426ded937f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.102475 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0496a42cfb744b25ad294d426ded937f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.105929 10837 tablet_service.cc:1467] Processing CreateTablet for tablet 0496a42cfb744b25ad294d426ded937f (DEFAULT_TABLE table=default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.106334 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0496a42cfb744b25ad294d426ded937f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.107046 11109 ts_tablet_manager.cc:1398] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.040s user 0.013s sys 0.000s
I20240501 14:02:27.107512 11109 tablet_bootstrap.cc:492] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.112095 11109 tablet_bootstrap.cc:654] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.113837 11039 heartbeater.cc:503] Master 127.9.224.189:42871 was elected leader, sending a full tablet report...
I20240501 14:02:27.116523 11110 ts_tablet_manager.cc:1398] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.049s user 0.012s sys 0.012s
I20240501 14:02:27.117070 11110 tablet_bootstrap.cc:492] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.118940 11110 tablet_bootstrap.cc:654] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.129508 11106 tablet_bootstrap.cc:492] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.131783 11106 tablet_bootstrap.cc:654] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.145946 11106 tablet_bootstrap.cc:492] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.146210 11106 ts_tablet_manager.cc:1367] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20240501 14:02:27.147176 11106 raft_consensus.cc:348] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.147512 11106 raft_consensus.cc:374] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.147603 11106 raft_consensus.cc:729] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.148008 11106 consensus_queue.cc:260] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.148720 11106 ts_tablet_manager.cc:1398] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:02:27.150316 10453 master_service.cc:946] ReplaceTablet: received request to replace tablet 6fe711480111456e967d9f41986b56bb from {username='slave'} at 127.0.0.1:59138
I20240501 14:02:27.156213 11109 tablet_bootstrap.cc:492] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.156435 11109 ts_tablet_manager.cc:1367] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.030s
I20240501 14:02:27.156594 11110 tablet_bootstrap.cc:492] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.156806 11110 ts_tablet_manager.cc:1367] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.015s
I20240501 14:02:27.157634 11109 raft_consensus.cc:348] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.157892 11109 raft_consensus.cc:374] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.158012 11109 raft_consensus.cc:729] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.158334 11109 consensus_queue.cc:260] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.158972 11109 ts_tablet_manager.cc:1398] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:27.158360 11110 raft_consensus.cc:348] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.160760 10452 catalog_manager.cc:2145] created HMS catalog entry for table [id=c425cd655f0e4e26b6a30c819c63f323]
I20240501 14:02:27.161377 10453 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: Sending DeleteTablet for 3 replicas of tablet 6fe711480111456e967d9f41986b56bb
I20240501 14:02:27.161684 11110 raft_consensus.cc:374] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.161808 11110 raft_consensus.cc:729] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.162099 11110 consensus_queue.cc:260] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.162564 10971 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe711480111456e967d9f41986b56bb with delete_type TABLET_DATA_DELETED (replaced by tablet b747db1c8b2b4921b3a1d23012f8b3a8) from {username='slave'} at 127.0.0.1:37170
I20240501 14:02:27.162972 11109 tablet_bootstrap.cc:492] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.164192 11110 ts_tablet_manager.cc:1398] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:02:27.164871 11110 tablet_bootstrap.cc:492] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.164881 11109 tablet_bootstrap.cc:654] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.166427 11110 tablet_bootstrap.cc:654] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.174768 11110 tablet_bootstrap.cc:492] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.175117 11110 ts_tablet_manager.cc:1367] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20240501 14:02:27.176020 11110 raft_consensus.cc:348] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.176383 11110 raft_consensus.cc:374] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.176513 11110 raft_consensus.cc:729] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.178378 11110 consensus_queue.cc:260] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.178977 10453 catalog_manager.cc:6554] ReplaceTablet: tablet 6fe711480111456e967d9f41986b56bb deleted and replaced by tablet b747db1c8b2b4921b3a1d23012f8b3a8
I20240501 14:02:27.179387 10453 master_service.cc:946] ReplaceTablet: received request to replace tablet 7a8f988937a24d3a9dd3ff7d4ec8296d from {username='slave'} at 127.0.0.1:59138
I20240501 14:02:27.179934 10837 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe711480111456e967d9f41986b56bb with delete_type TABLET_DATA_DELETED (replaced by tablet b747db1c8b2b4921b3a1d23012f8b3a8) from {username='slave'} at 127.0.0.1:46470
I20240501 14:02:27.179888 10452 master_service.cc:946] ReplaceTablet: received request to replace tablet bb38cc63d5714e2b96371278ddc8e804 from {username='slave'} at 127.0.0.1:59138
I20240501 14:02:27.184211 10698 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe711480111456e967d9f41986b56bb with delete_type TABLET_DATA_DELETED (replaced by tablet b747db1c8b2b4921b3a1d23012f8b3a8) from {username='slave'} at 127.0.0.1:60726
I20240501 14:02:27.189473 11110 ts_tablet_manager.cc:1398] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240501 14:02:27.194586 11110 tablet_bootstrap.cc:492] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.196738 11110 tablet_bootstrap.cc:654] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.196825 10697 tablet_service.cc:1467] Processing CreateTablet for tablet dfd4544926c04f7fa34419a8108e8d1e (DEFAULT_TABLE table=default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.197355 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd4544926c04f7fa34419a8108e8d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.197759 10838 tablet_service.cc:1467] Processing CreateTablet for tablet dfd4544926c04f7fa34419a8108e8d1e (DEFAULT_TABLE table=default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.198236 10838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd4544926c04f7fa34419a8108e8d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.200245 11125 tablet_replica.cc:330] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: stopping tablet replica
I20240501 14:02:27.200816 11125 raft_consensus.cc:2229] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:27.201282 11125 raft_consensus.cc:2258] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:27.201655 10836 tablet_service.cc:1467] Processing CreateTablet for tablet b747db1c8b2b4921b3a1d23012f8b3a8 (DEFAULT_TABLE table=default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.202034 10700 tablet_service.cc:1467] Processing CreateTablet for tablet b747db1c8b2b4921b3a1d23012f8b3a8 (DEFAULT_TABLE table=default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.202224 10836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b747db1c8b2b4921b3a1d23012f8b3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.202500 10700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b747db1c8b2b4921b3a1d23012f8b3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.205999 10969 tablet_service.cc:1467] Processing CreateTablet for tablet dfd4544926c04f7fa34419a8108e8d1e (DEFAULT_TABLE table=default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.206501 10969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd4544926c04f7fa34419a8108e8d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.210479 11106 tablet_bootstrap.cc:492] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.212215 11106 tablet_bootstrap.cc:654] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.213089 10972 tablet_service.cc:1467] Processing CreateTablet for tablet b747db1c8b2b4921b3a1d23012f8b3a8 (DEFAULT_TABLE table=default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.213541 10972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b747db1c8b2b4921b3a1d23012f8b3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.221529 11125 ts_tablet_manager.cc:1875] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:02:27.221583 11126 tablet_replica.cc:330] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: stopping tablet replica
I20240501 14:02:27.222324 11126 raft_consensus.cc:2229] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:27.222868 11126 raft_consensus.cc:2258] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:27.224586 11126 ts_tablet_manager.cc:1875] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:02:27.228268 10452 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: Sending DeleteTablet for 3 replicas of tablet bb38cc63d5714e2b96371278ddc8e804
I20240501 14:02:27.229003 10452 catalog_manager.cc:6554] ReplaceTablet: tablet bb38cc63d5714e2b96371278ddc8e804 deleted and replaced by tablet 543d48b213be415c9cd17bbf67e0bb5d
I20240501 14:02:27.229439 10698 tablet_service.cc:1514] Processing DeleteTablet for tablet bb38cc63d5714e2b96371278ddc8e804 with delete_type TABLET_DATA_DELETED (replaced by tablet 543d48b213be415c9cd17bbf67e0bb5d) from {username='slave'} at 127.0.0.1:60726
I20240501 14:02:27.229506 10837 tablet_service.cc:1514] Processing DeleteTablet for tablet bb38cc63d5714e2b96371278ddc8e804 with delete_type TABLET_DATA_DELETED (replaced by tablet 543d48b213be415c9cd17bbf67e0bb5d) from {username='slave'} at 127.0.0.1:46470
I20240501 14:02:27.230125 10972 tablet_service.cc:1514] Processing DeleteTablet for tablet bb38cc63d5714e2b96371278ddc8e804 with delete_type TABLET_DATA_DELETED (replaced by tablet 543d48b213be415c9cd17bbf67e0bb5d) from {username='slave'} at 127.0.0.1:37170
I20240501 14:02:27.234192 11125 ts_tablet_manager.cc:1888] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 14:02:27.234706 11125 log.cc:1196] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: Deleting WAL directory at /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal/wals/6fe711480111456e967d9f41986b56bb
I20240501 14:02:27.235532 11125 ts_tablet_manager.cc:1909] T 6fe711480111456e967d9f41986b56bb P 156c3e56589d4969b853b3b512301a48: Deleting consensus metadata
I20240501 14:02:27.236631 11126 ts_tablet_manager.cc:1888] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 14:02:27.236800 11126 log.cc:1196] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: Deleting WAL directory at /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal/wals/6fe711480111456e967d9f41986b56bb
I20240501 14:02:27.237327 11126 ts_tablet_manager.cc:1909] T 6fe711480111456e967d9f41986b56bb P 5e09841056a14c29be2546cc1d0b4643: Deleting consensus metadata
I20240501 14:02:27.238981 10444 catalog_manager.cc:4830] TS 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539): tablet 6fe711480111456e967d9f41986b56bb (table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]) successfully deleted
I20240501 14:02:27.239284 10445 catalog_manager.cc:4830] TS 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403): tablet 6fe711480111456e967d9f41986b56bb (table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]) successfully deleted
I20240501 14:02:27.240108 10453 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e048fbcc4e544adea37b6bd9e5ac8543: Sending DeleteTablet for 3 replicas of tablet 7a8f988937a24d3a9dd3ff7d4ec8296d
I20240501 14:02:27.240517 11125 tablet_replica.cc:330] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: stopping tablet replica
I20240501 14:02:27.240684 10453 catalog_manager.cc:6554] ReplaceTablet: tablet 7a8f988937a24d3a9dd3ff7d4ec8296d deleted and replaced by tablet 37ae9258fe314b9b9271a8f25a085be2
I20240501 14:02:27.241006 10698 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a8f988937a24d3a9dd3ff7d4ec8296d with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae9258fe314b9b9271a8f25a085be2) from {username='slave'} at 127.0.0.1:60726
I20240501 14:02:27.241343 11125 raft_consensus.cc:2229] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:27.241680 11125 raft_consensus.cc:2258] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:27.242848 11125 ts_tablet_manager.cc:1875] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:02:27.243220 10453 master_service.cc:946] ReplaceTablet: received request to replace tablet 6fe711480111456e967d9f41986b56bb from {username='slave'} at 127.0.0.1:59138
I20240501 14:02:27.243727 10453 master_service.cc:946] ReplaceTablet: received request to replace tablet bb38cc63d5714e2b96371278ddc8e804 from {username='slave'} at 127.0.0.1:59138
I20240501 14:02:27.244254 10453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_4801897b969f4fb8a4d46503057f4169"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:27.245548 10972 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a8f988937a24d3a9dd3ff7d4ec8296d with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae9258fe314b9b9271a8f25a085be2) from {username='slave'} at 127.0.0.1:37170
I20240501 14:02:27.255704 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 543d48b213be415c9cd17bbf67e0bb5d (DEFAULT_TABLE table=default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.256237 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543d48b213be415c9cd17bbf67e0bb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.258246 11125 ts_tablet_manager.cc:1888] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 14:02:27.259495 11125 log.cc:1196] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: Deleting WAL directory at /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal/wals/bb38cc63d5714e2b96371278ddc8e804
I20240501 14:02:27.260156 11125 ts_tablet_manager.cc:1909] T bb38cc63d5714e2b96371278ddc8e804 P 156c3e56589d4969b853b3b512301a48: Deleting consensus metadata
I20240501 14:02:27.264344 10837 tablet_service.cc:1467] Processing CreateTablet for tablet 543d48b213be415c9cd17bbf67e0bb5d (DEFAULT_TABLE table=default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.264524 10445 catalog_manager.cc:4830] TS 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403): tablet bb38cc63d5714e2b96371278ddc8e804 (table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]) successfully deleted
I20240501 14:02:27.264851 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543d48b213be415c9cd17bbf67e0bb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.267028 11126 tablet_replica.cc:330] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: stopping tablet replica
I20240501 14:02:27.267233 11126 raft_consensus.cc:2229] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:27.267411 11126 raft_consensus.cc:2258] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240501 14:02:27.267479 10444 catalog_manager.cc:4810] TS 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539): delete failed for tablet bb38cc63d5714e2b96371278ddc8e804 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb38cc63d5714e2b96371278ddc8e804 already in progress: creating tablet
I20240501 14:02:27.268189 11126 ts_tablet_manager.cc:1875] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:02:27.268095 10452 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59138:
table { table_name: "default.Table_cc9ce1d1031f42789771ae5eb4a6df43" } new_table_name: "default.Table_b8695d7d9f484f03995f17c0bbc61ee3" modify_external_catalogs: true
I20240501 14:02:27.268571 10836 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a8f988937a24d3a9dd3ff7d4ec8296d with delete_type TABLET_DATA_DELETED (replaced by tablet 37ae9258fe314b9b9271a8f25a085be2) from {username='slave'} at 127.0.0.1:46470
I20240501 14:02:27.269515 10972 tablet_service.cc:1467] Processing CreateTablet for tablet 543d48b213be415c9cd17bbf67e0bb5d (DEFAULT_TABLE table=default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.269950 10972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543d48b213be415c9cd17bbf67e0bb5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:02:27.273917 11040 tablet.cc:2369] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:27.274219 11126 ts_tablet_manager.cc:1888] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 14:02:27.274360 11126 log.cc:1196] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: Deleting WAL directory at /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal/wals/7a8f988937a24d3a9dd3ff7d4ec8296d
I20240501 14:02:27.274719 11126 ts_tablet_manager.cc:1909] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 5e09841056a14c29be2546cc1d0b4643: Deleting consensus metadata
I20240501 14:02:27.275887 10444 catalog_manager.cc:4830] TS 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539): tablet 7a8f988937a24d3a9dd3ff7d4ec8296d (table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]) successfully deleted
I20240501 14:02:27.279044 11125 tablet_replica.cc:330] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: stopping tablet replica
I20240501 14:02:27.279311 11125 raft_consensus.cc:2229] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:27.279619 11125 raft_consensus.cc:2258] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:27.280483 11125 ts_tablet_manager.cc:1875] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:02:27.287236 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 37ae9258fe314b9b9271a8f25a085be2 (DEFAULT_TABLE table=default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.287743 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ae9258fe314b9b9271a8f25a085be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.291787 10837 tablet_service.cc:1467] Processing CreateTablet for tablet 37ae9258fe314b9b9271a8f25a085be2 (DEFAULT_TABLE table=default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.292486 10969 tablet_service.cc:1467] Processing CreateTablet for tablet 37ae9258fe314b9b9271a8f25a085be2 (DEFAULT_TABLE table=default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.292878 10969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ae9258fe314b9b9271a8f25a085be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.293267 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ae9258fe314b9b9271a8f25a085be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.299082 11125 ts_tablet_manager.cc:1888] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 14:02:27.299223 11125 log.cc:1196] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: Deleting WAL directory at /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-1/wal/wals/7a8f988937a24d3a9dd3ff7d4ec8296d
I20240501 14:02:27.299551 11125 ts_tablet_manager.cc:1909] T 7a8f988937a24d3a9dd3ff7d4ec8296d P 156c3e56589d4969b853b3b512301a48: Deleting consensus metadata
I20240501 14:02:27.300611 10445 catalog_manager.cc:4830] TS 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403): tablet 7a8f988937a24d3a9dd3ff7d4ec8296d (table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]) successfully deleted
I20240501 14:02:27.302194 10458 catalog_manager.cc:2145] created HMS catalog entry for table [id=35433d009fbb4c27b7fa21916b08ddfd]
I20240501 14:02:27.313220 10458 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_f4862f63c8104085837aee11c33c2718"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:27.319471 11122 tablet_replica.cc:330] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: stopping tablet replica
I20240501 14:02:27.320215 11122 raft_consensus.cc:2229] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:27.320755 11122 raft_consensus.cc:2258] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:27.326941 11110 tablet_bootstrap.cc:492] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.327111 11110 ts_tablet_manager.cc:1367] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.133s user 0.013s sys 0.033s
I20240501 14:02:27.327903 11110 raft_consensus.cc:348] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.328110 11110 raft_consensus.cc:374] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.328186 11110 raft_consensus.cc:729] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.328451 11110 consensus_queue.cc:260] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.330363 11122 ts_tablet_manager.cc:1875] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:02:27.332674 11110 ts_tablet_manager.cc:1398] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:02:27.335284 11106 tablet_bootstrap.cc:492] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.335466 11106 ts_tablet_manager.cc:1367] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.043s
I20240501 14:02:27.335877 11110 tablet_bootstrap.cc:492] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.336230 11106 raft_consensus.cc:348] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.336441 11106 raft_consensus.cc:374] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.336544 11106 raft_consensus.cc:729] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.336784 10697 tablet_service.cc:1467] Processing CreateTablet for tablet fd6038c451a846c6b6541c0ac115ca12 (DEFAULT_TABLE table=default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.336773 11106 consensus_queue.cc:260] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.337224 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6038c451a846c6b6541c0ac115ca12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.337291 11106 ts_tablet_manager.cc:1398] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:27.337620 11110 tablet_bootstrap.cc:654] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.338660 11106 tablet_bootstrap.cc:492] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.340176 11106 tablet_bootstrap.cc:654] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.347236 11110 tablet_bootstrap.cc:492] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.347515 11110 ts_tablet_manager.cc:1367] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20240501 14:02:27.348281 11110 raft_consensus.cc:348] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.348526 11110 raft_consensus.cc:374] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.348604 11110 raft_consensus.cc:729] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.348852 11110 consensus_queue.cc:260] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.349684 10969 tablet_service.cc:1467] Processing CreateTablet for tablet fd6038c451a846c6b6541c0ac115ca12 (DEFAULT_TABLE table=default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.349975 11109 tablet_bootstrap.cc:492] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.350067 10969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6038c451a846c6b6541c0ac115ca12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.350167 11109 ts_tablet_manager.cc:1367] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.187s user 0.024s sys 0.070s
I20240501 14:02:27.350104 10837 tablet_service.cc:1467] Processing CreateTablet for tablet fd6038c451a846c6b6541c0ac115ca12 (DEFAULT_TABLE table=default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.350498 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6038c451a846c6b6541c0ac115ca12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.350996 11109 raft_consensus.cc:348] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.351233 11109 raft_consensus.cc:374] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.351302 11109 raft_consensus.cc:729] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.351619 11109 consensus_queue.cc:260] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.354310 11110 ts_tablet_manager.cc:1398] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:02:27.355474 11110 tablet_bootstrap.cc:492] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.355715 11109 ts_tablet_manager.cc:1398] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:02:27.357017 11110 tablet_bootstrap.cc:654] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.357075 11109 tablet_bootstrap.cc:492] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.358672 11109 tablet_bootstrap.cc:654] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.365746 11106 tablet_bootstrap.cc:492] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.365947 11106 ts_tablet_manager.cc:1367] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20240501 14:02:27.366779 11106 raft_consensus.cc:348] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.367007 11106 raft_consensus.cc:374] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.367081 11106 raft_consensus.cc:729] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.367322 11106 consensus_queue.cc:260] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.367894 11106 ts_tablet_manager.cc:1398] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:27.368191 11106 tablet_bootstrap.cc:492] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.369892 11106 tablet_bootstrap.cc:654] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.371697 11109 tablet_bootstrap.cc:492] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.371852 11109 ts_tablet_manager.cc:1367] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240501 14:02:27.372577 11109 raft_consensus.cc:348] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.372797 11109 raft_consensus.cc:374] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.372941 11109 raft_consensus.cc:729] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.373296 11122 ts_tablet_manager.cc:1888] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 14:02:27.373242 11109 consensus_queue.cc:260] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.375468 11109 ts_tablet_manager.cc:1398] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:27.375864 11109 tablet_bootstrap.cc:492] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.377673 11109 tablet_bootstrap.cc:654] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.378260 11108 raft_consensus.cc:482] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.378495 11108 raft_consensus.cc:504] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.380499 11108 leader_election.cc:288] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.382653 11122 log.cc:1196] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: Deleting WAL directory at /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal/wals/6fe711480111456e967d9f41986b56bb
I20240501 14:02:27.383359 11106 tablet_bootstrap.cc:492] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.383431 11122 ts_tablet_manager.cc:1909] T 6fe711480111456e967d9f41986b56bb P c1d9952ad6a24cecbcde3d02a59c9a75: Deleting consensus metadata
I20240501 14:02:27.383718 11106 ts_tablet_manager.cc:1367] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20240501 14:02:27.384763 11106 raft_consensus.cc:348] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.385069 11106 raft_consensus.cc:374] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.385206 11106 raft_consensus.cc:729] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.385473 11106 consensus_queue.cc:260] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.387503 11106 ts_tablet_manager.cc:1398] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:02:27.387854 11106 tablet_bootstrap.cc:492] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.379544 11110 tablet_bootstrap.cc:492] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.389253 11110 ts_tablet_manager.cc:1367] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.006s
I20240501 14:02:27.389741 11106 tablet_bootstrap.cc:654] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.391333 11110 raft_consensus.cc:348] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.391583 11110 raft_consensus.cc:374] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.391670 11110 raft_consensus.cc:729] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.392457 11110 consensus_queue.cc:260] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.394640 11110 ts_tablet_manager.cc:1398] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:02:27.398285 11110 tablet_bootstrap.cc:492] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.400005 11110 tablet_bootstrap.cc:654] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.404017 10445 catalog_manager.cc:4830] TS c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371): tablet 6fe711480111456e967d9f41986b56bb (table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]) successfully deleted
I20240501 14:02:27.404501 11122 tablet_replica.cc:330] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: stopping tablet replica
I20240501 14:02:27.404745 11122 raft_consensus.cc:2229] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:27.408404 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd4544926c04f7fa34419a8108e8d1e" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643" is_pre_election: true
W20240501 14:02:27.409752 10793 leader_election.cc:341] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:27.411392 11109 tablet_bootstrap.cc:492] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.411604 11109 ts_tablet_manager.cc:1367] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.001s
I20240501 14:02:27.412422 11109 raft_consensus.cc:348] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.412691 11109 raft_consensus.cc:374] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.412766 11109 raft_consensus.cc:729] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.404959 11122 raft_consensus.cc:2258] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:27.413076 11109 consensus_queue.cc:260] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.414141 11122 ts_tablet_manager.cc:1875] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:02:27.415755 11109 ts_tablet_manager.cc:1398] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:27.417098 11109 tablet_bootstrap.cc:492] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.417706 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd4544926c04f7fa34419a8108e8d1e" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" is_pre_election: true
I20240501 14:02:27.418421 10992 raft_consensus.cc:2454] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 0.
I20240501 14:02:27.419116 11109 tablet_bootstrap.cc:654] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.419507 10794 leader_election.cc:302] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters: 5e09841056a14c29be2546cc1d0b4643
I20240501 14:02:27.420161 11108 raft_consensus.cc:2781] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:27.420321 11108 raft_consensus.cc:482] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:27.420742 11108 raft_consensus.cc:3037] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.424422 11108 raft_consensus.cc:504] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.426249 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd4544926c04f7fa34419a8108e8d1e" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643"
W20240501 14:02:27.427022 10793 leader_election.cc:341] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:27.427456 11108 leader_election.cc:288] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 election: Requested vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.428728 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd4544926c04f7fa34419a8108e8d1e" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75"
I20240501 14:02:27.428997 10992 raft_consensus.cc:3037] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.432543 10992 raft_consensus.cc:2454] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 1.
I20240501 14:02:27.435770 10794 leader_election.cc:302] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters: 5e09841056a14c29be2546cc1d0b4643
I20240501 14:02:27.436687 11108 raft_consensus.cc:2781] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:27.444373 11108 raft_consensus.cc:686] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [term 1 LEADER]: Becoming Leader. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Running, Role: LEADER
I20240501 14:02:27.445125 11108 consensus_queue.cc:237] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.460155 11106 tablet_bootstrap.cc:492] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.460341 11106 ts_tablet_manager.cc:1367] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.073s user 0.029s sys 0.036s
I20240501 14:02:27.460717 11110 tablet_bootstrap.cc:492] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.460896 11110 ts_tablet_manager.cc:1367] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.024s
I20240501 14:02:27.461150 11106 raft_consensus.cc:348] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.461403 11106 raft_consensus.cc:374] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.461477 11106 raft_consensus.cc:729] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.461716 11106 consensus_queue.cc:260] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.461747 11110 raft_consensus.cc:348] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.461961 11110 raft_consensus.cc:374] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.462038 11110 raft_consensus.cc:729] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.462226 11106 ts_tablet_manager.cc:1398] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:27.462250 11110 consensus_queue.cc:260] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.462539 11106 tablet_bootstrap.cc:492] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.464063 11106 tablet_bootstrap.cc:654] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.464478 11122 ts_tablet_manager.cc:1888] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 14:02:27.464613 11122 log.cc:1196] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: Deleting WAL directory at /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal/wals/bb38cc63d5714e2b96371278ddc8e804
I20240501 14:02:27.464967 11122 ts_tablet_manager.cc:1909] T bb38cc63d5714e2b96371278ddc8e804 P c1d9952ad6a24cecbcde3d02a59c9a75: Deleting consensus metadata
I20240501 14:02:27.465840 11110 ts_tablet_manager.cc:1398] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:02:27.469135 11110 tablet_bootstrap.cc:492] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.470000 10445 catalog_manager.cc:4830] TS c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371): tablet bb38cc63d5714e2b96371278ddc8e804 (table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]) successfully deleted
I20240501 14:02:27.470531 11109 tablet_bootstrap.cc:492] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.470530 11106 tablet_bootstrap.cc:492] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.470705 11109 ts_tablet_manager.cc:1367] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.009s
I20240501 14:02:27.470799 11110 tablet_bootstrap.cc:654] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.470813 11106 ts_tablet_manager.cc:1367] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240501 14:02:27.471537 11109 raft_consensus.cc:348] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.471537 11106 raft_consensus.cc:348] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.471765 11106 raft_consensus.cc:374] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.471765 11109 raft_consensus.cc:374] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.471859 11109 raft_consensus.cc:729] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.471863 11106 raft_consensus.cc:729] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.472098 11106 consensus_queue.cc:260] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.472121 11109 consensus_queue.cc:260] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.472558 11110 tablet_bootstrap.cc:492] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.472712 11110 ts_tablet_manager.cc:1367] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:27.472733 11106 ts_tablet_manager.cc:1398] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:27.473555 11110 raft_consensus.cc:348] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.473762 11110 raft_consensus.cc:374] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.473846 11110 raft_consensus.cc:729] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.474068 11110 consensus_queue.cc:260] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.475657 11110 ts_tablet_manager.cc:1398] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:27.475881 11109 ts_tablet_manager.cc:1398] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240501 14:02:27.476218 11109 tablet_bootstrap.cc:492] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.477706 11110 tablet_bootstrap.cc:492] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.480547 11110 tablet_bootstrap.cc:654] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.482169 11109 tablet_bootstrap.cc:654] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.482506 11122 tablet_replica.cc:330] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: stopping tablet replica
I20240501 14:02:27.482779 11122 raft_consensus.cc:2229] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:27.483100 11122 raft_consensus.cc:2258] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:27.484138 11122 ts_tablet_manager.cc:1875] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:02:27.488714 11110 tablet_bootstrap.cc:492] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.488925 11110 ts_tablet_manager.cc:1367] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 14:02:27.489830 11110 raft_consensus.cc:348] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.490070 11110 raft_consensus.cc:374] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.490150 11110 raft_consensus.cc:729] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.490447 11110 consensus_queue.cc:260] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.492069 11122 ts_tablet_manager.cc:1888] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 14:02:27.492250 11122 log.cc:1196] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: Deleting WAL directory at /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-2/wal/wals/7a8f988937a24d3a9dd3ff7d4ec8296d
I20240501 14:02:27.492729 11122 ts_tablet_manager.cc:1909] T 7a8f988937a24d3a9dd3ff7d4ec8296d P c1d9952ad6a24cecbcde3d02a59c9a75: Deleting consensus metadata
I20240501 14:02:27.494179 10445 catalog_manager.cc:4830] TS c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371): tablet 7a8f988937a24d3a9dd3ff7d4ec8296d (table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]) successfully deleted
I20240501 14:02:27.495100 11110 ts_tablet_manager.cc:1398] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:02:27.496855 10454 catalog_manager.cc:2145] created HMS catalog entry for table [id=3a6f7d0bf2794a7e8717c2355971c0a5]
I20240501 14:02:27.499742 11110 tablet_bootstrap.cc:492] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.501415 11110 tablet_bootstrap.cc:654] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.503583 11109 tablet_bootstrap.cc:492] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.503814 11109 ts_tablet_manager.cc:1367] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.007s
I20240501 14:02:27.504722 11109 raft_consensus.cc:348] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.504959 11109 raft_consensus.cc:374] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.505071 11109 raft_consensus.cc:729] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.505373 11109 consensus_queue.cc:260] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.506361 11109 ts_tablet_manager.cc:1398] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:02:27.506691 11109 tablet_bootstrap.cc:492] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.506706 10969 tablet_service.cc:1514] Processing DeleteTablet for tablet bb38cc63d5714e2b96371278ddc8e804 with delete_type TABLET_DATA_DELETED (replaced by tablet 543d48b213be415c9cd17bbf67e0bb5d) from {username='slave'} at 127.0.0.1:37170
I20240501 14:02:27.508502 11109 tablet_bootstrap.cc:654] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
W20240501 14:02:27.509207 10445 catalog_manager.cc:4797] TS c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371): delete failed for tablet bb38cc63d5714e2b96371278ddc8e804 because the tablet was not found. No further retry: Not found: Tablet not found: bb38cc63d5714e2b96371278ddc8e804
I20240501 14:02:27.518734 10969 tablet_service.cc:1467] Processing CreateTablet for tablet 552cf2dae53e4e81aad0bf5453eef53d (DEFAULT_TABLE table=default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.518841 10837 tablet_service.cc:1467] Processing CreateTablet for tablet 552cf2dae53e4e81aad0bf5453eef53d (DEFAULT_TABLE table=default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.519217 10969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552cf2dae53e4e81aad0bf5453eef53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.519280 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552cf2dae53e4e81aad0bf5453eef53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.524749 10454 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "default.Table_351f54255b01483daad53a39989a377c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:27.527305 11106 tablet_bootstrap.cc:492] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.528299 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 552cf2dae53e4e81aad0bf5453eef53d (DEFAULT_TABLE table=default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.529044 11106 tablet_bootstrap.cc:654] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.533288 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552cf2dae53e4e81aad0bf5453eef53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.535012 11118 raft_consensus.cc:482] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.535717 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--webserver_interface=127.9.224.188
--webserver_port=35625
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240501 14:02:27.535280 11118 raft_consensus.cc:504] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.538285 11118 leader_election.cc:288] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.546104 11110 tablet_bootstrap.cc:492] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.546111 11109 tablet_bootstrap.cc:492] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.546299 11110 ts_tablet_manager.cc:1367] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.005s
I20240501 14:02:27.546311 11109 ts_tablet_manager.cc:1367] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.000s
I20240501 14:02:27.547231 11109 raft_consensus.cc:348] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.547217 11110 raft_consensus.cc:348] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.547477 11109 raft_consensus.cc:374] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.547482 11110 raft_consensus.cc:374] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.547564 11110 raft_consensus.cc:729] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.547571 11109 raft_consensus.cc:729] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.547834 11109 consensus_queue.cc:260] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.547927 11110 consensus_queue.cc:260] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.548457 11109 ts_tablet_manager.cc:1398] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:27.549132 11109 tablet_bootstrap.cc:492] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.550523 11110 ts_tablet_manager.cc:1398] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:02:27.550706 10455 catalog_manager.cc:2145] created HMS catalog entry for table [id=ee51129ad60e40bcb0a839c84a33a5e8]
I20240501 14:02:27.553947 11110 tablet_bootstrap.cc:492] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.559703 11109 tablet_bootstrap.cc:654] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.564460 11110 tablet_bootstrap.cc:654] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.568501 10969 tablet_service.cc:1467] Processing CreateTablet for tablet 56ff9d28dcf24f24935f5c739316ff70 (DEFAULT_TABLE table=default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.568956 10969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ff9d28dcf24f24935f5c739316ff70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.568620 10455 catalog_manager.cc:5484] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 156c3e56589d4969b853b3b512301a48 (127.9.224.130). New cstate: current_term: 1 leader_uuid: "156c3e56589d4969b853b3b512301a48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:27.569757 11106 tablet_bootstrap.cc:492] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.569938 11106 ts_tablet_manager.cc:1367] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.010s
I20240501 14:02:27.570757 11106 raft_consensus.cc:348] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.571012 11106 raft_consensus.cc:374] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.571111 11106 raft_consensus.cc:729] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.571058 10837 tablet_service.cc:1467] Processing CreateTablet for tablet 56ff9d28dcf24f24935f5c739316ff70 (DEFAULT_TABLE table=default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.571473 10837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ff9d28dcf24f24935f5c739316ff70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.571396 11106 consensus_queue.cc:260] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.574906 11106 ts_tablet_manager.cc:1398] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:02:27.577595 11106 tablet_bootstrap.cc:492] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48: Bootstrap starting.
I20240501 14:02:27.578141 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 56ff9d28dcf24f24935f5c739316ff70 (DEFAULT_TABLE table=default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:27.578703 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ff9d28dcf24f24935f5c739316ff70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:27.579171 11106 tablet_bootstrap.cc:654] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.579617 11116 raft_consensus.cc:482] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.579788 11116 raft_consensus.cc:504] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.581808 11116 leader_election.cc:288] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:27.583937 10697 tablet_service.cc:1514] Processing DeleteTablet for tablet bb38cc63d5714e2b96371278ddc8e804 with delete_type TABLET_DATA_DELETED (replaced by tablet 543d48b213be415c9cd17bbf67e0bb5d) from {username='slave'} at 127.0.0.1:60726
I20240501 14:02:27.584345 11126 tablet_replica.cc:330] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: stopping tablet replica
I20240501 14:02:27.584555 11126 raft_consensus.cc:2229] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:27.584744 11126 raft_consensus.cc:2258] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:27.585624 11126 ts_tablet_manager.cc:1875] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:02:27.592029 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0496a42cfb744b25ad294d426ded937f" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643" is_pre_election: true
I20240501 14:02:27.592350 10720 raft_consensus.cc:2454] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 0.
I20240501 14:02:27.593008 10927 leader_election.cc:302] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e09841056a14c29be2546cc1d0b4643, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:27.593616 11116 raft_consensus.cc:2781] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:27.593736 11116 raft_consensus.cc:482] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:27.593827 11116 raft_consensus.cc:3037] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:02:27.594056 10656 leader_election.cc:341] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403): Not found: Tablet not found: bb38cc63d5714e2b96371278ddc8e804
I20240501 14:02:27.594808 11126 ts_tablet_manager.cc:1888] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 14:02:27.594928 11126 log.cc:1196] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: Deleting WAL directory at /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/ts-0/wal/wals/bb38cc63d5714e2b96371278ddc8e804
I20240501 14:02:27.595335 11126 ts_tablet_manager.cc:1909] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: Deleting consensus metadata
I20240501 14:02:27.596072 10444 catalog_manager.cc:4830] TS 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539): tablet bb38cc63d5714e2b96371278ddc8e804 (table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]) successfully deleted
W20240501 14:02:27.600132 10768 tablet.cc:2369] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:27.600266 11106 tablet_bootstrap.cc:492] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48: No bootstrap required, opened a new log
I20240501 14:02:27.600438 11106 ts_tablet_manager.cc:1367] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20240501 14:02:27.601286 11106 raft_consensus.cc:348] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.601524 11106 raft_consensus.cc:374] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.601636 11106 raft_consensus.cc:729] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.601298 11116 raft_consensus.cc:504] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.601924 11106 consensus_queue.cc:260] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.603359 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0496a42cfb744b25ad294d426ded937f" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643"
I20240501 14:02:27.603616 10720 raft_consensus.cc:3037] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.605656 11106 ts_tablet_manager.cc:1398] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:02:27.608623 11108 raft_consensus.cc:482] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.608800 11108 raft_consensus.cc:504] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.609537 11108 leader_election.cc:288] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.610543 10720 raft_consensus.cc:2454] T 0496a42cfb744b25ad294d426ded937f P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 1.
W20240501 14:02:27.610903 10793 leader_election.cc:341] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539): Not found: Tablet not found: 56ff9d28dcf24f24935f5c739316ff70
I20240501 14:02:27.611093 10927 leader_election.cc:302] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e09841056a14c29be2546cc1d0b4643, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:27.617408 10455 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59138:
table { table_name: "default.Table_d3a10d5d2bfb4854bd055ff3e0768f4e" } modify_external_catalogs: true
I20240501 14:02:27.617702 10455 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59138:
table { table_name: "default.Table_d3a10d5d2bfb4854bd055ff3e0768f4e" } modify_external_catalogs: true
I20240501 14:02:27.621244 11110 tablet_bootstrap.cc:492] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.621443 11110 ts_tablet_manager.cc:1367] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.068s user 0.014s sys 0.014s
I20240501 14:02:27.622272 11110 raft_consensus.cc:348] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.622511 11110 raft_consensus.cc:374] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.622586 11110 raft_consensus.cc:729] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.622830 11110 consensus_queue.cc:260] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.624840 11116 leader_election.cc:288] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 election: Requested vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:27.625166 11116 raft_consensus.cc:2781] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:27.625375 11116 raft_consensus.cc:686] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 LEADER]: Becoming Leader. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Running, Role: LEADER
I20240501 14:02:27.626025 11116 consensus_queue.cc:237] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
W20240501 14:02:27.662019 10927 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.224.129:38539, user_credentials={real_user=slave}} blocked reactor thread for 50901.7us
I20240501 14:02:27.662942 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ff9d28dcf24f24935f5c739316ff70" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" is_pre_election: true
W20240501 14:02:27.663829 10794 leader_election.cc:341] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:27.663995 10794 leader_election.cc:302] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 156c3e56589d4969b853b3b512301a48; no voters: 5e09841056a14c29be2546cc1d0b4643, c1d9952ad6a24cecbcde3d02a59c9a75
I20240501 14:02:27.664499 11108 raft_consensus.cc:2726] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:27.666322 11109 tablet_bootstrap.cc:492] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.666644 11109 ts_tablet_manager.cc:1367] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.118s user 0.013s sys 0.025s
I20240501 14:02:27.667578 11109 raft_consensus.cc:348] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.667914 11109 raft_consensus.cc:374] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.668006 11109 raft_consensus.cc:729] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.668316 11109 consensus_queue.cc:260] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
W20240501 14:02:27.669934 10656 leader_election.cc:341] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371): Not found: Tablet not found: bb38cc63d5714e2b96371278ddc8e804
I20240501 14:02:27.670060 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0496a42cfb744b25ad294d426ded937f" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48"
I20240501 14:02:27.670120 10656 leader_election.cc:302] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e09841056a14c29be2546cc1d0b4643; no voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75
I20240501 14:02:27.669979 11109 ts_tablet_manager.cc:1398] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:02:27.670433 10857 raft_consensus.cc:3037] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:02:27.670483 10656 raft_consensus.cc:2670] T bb38cc63d5714e2b96371278ddc8e804 P 5e09841056a14c29be2546cc1d0b4643: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240501 14:02:27.670733 11109 tablet_bootstrap.cc:492] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.671588 10858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0496a42cfb744b25ad294d426ded937f" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48" is_pre_election: true
I20240501 14:02:27.672480 11109 tablet_bootstrap.cc:654] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.673724 10857 raft_consensus.cc:2454] T 0496a42cfb744b25ad294d426ded937f P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 1.
I20240501 14:02:27.681612 11110 ts_tablet_manager.cc:1398] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.060s user 0.016s sys 0.039s
I20240501 14:02:27.682085 11110 tablet_bootstrap.cc:492] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75: Bootstrap starting.
I20240501 14:02:27.683925 11110 tablet_bootstrap.cc:654] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.687664 11109 tablet_bootstrap.cc:492] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.687845 11109 ts_tablet_manager.cc:1367] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20240501 14:02:27.688669 11109 raft_consensus.cc:348] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.688905 11109 raft_consensus.cc:374] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.689014 11109 raft_consensus.cc:729] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.689291 11109 consensus_queue.cc:260] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.692301 11109 ts_tablet_manager.cc:1398] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:02:27.694325 11110 tablet_bootstrap.cc:492] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75: No bootstrap required, opened a new log
I20240501 14:02:27.694497 11110 ts_tablet_manager.cc:1367] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240501 14:02:27.694882 11109 tablet_bootstrap.cc:492] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.695281 11110 raft_consensus.cc:348] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.695497 11110 raft_consensus.cc:374] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.695585 11110 raft_consensus.cc:729] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.695626 11118 raft_consensus.cc:482] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.695773 11118 raft_consensus.cc:504] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.695801 11110 consensus_queue.cc:260] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.696419 11118 leader_election.cc:288] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:27.696911 11110 ts_tablet_manager.cc:1398] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:27.697911 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd4544926c04f7fa34419a8108e8d1e" candidate_uuid: "5e09841056a14c29be2546cc1d0b4643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" is_pre_election: true
I20240501 14:02:27.698129 10992 raft_consensus.cc:2379] T dfd4544926c04f7fa34419a8108e8d1e P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e09841056a14c29be2546cc1d0b4643 in current term 1: Already voted for candidate 156c3e56589d4969b853b3b512301a48 in this term.
I20240501 14:02:27.698617 11159 raft_consensus.cc:482] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.698743 11159 raft_consensus.cc:504] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.699477 11159 leader_election.cc:288] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:27.700084 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552cf2dae53e4e81aad0bf5453eef53d" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643" is_pre_election: true
I20240501 14:02:27.700409 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd4544926c04f7fa34419a8108e8d1e" candidate_uuid: "5e09841056a14c29be2546cc1d0b4643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48" is_pre_election: true
W20240501 14:02:27.700774 10927 leader_election.cc:341] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:27.702311 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552cf2dae53e4e81aad0bf5453eef53d" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48" is_pre_election: true
I20240501 14:02:27.702553 10857 raft_consensus.cc:2454] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 0.
I20240501 14:02:27.702701 11109 tablet_bootstrap.cc:654] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.703855 10656 leader_election.cc:302] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e09841056a14c29be2546cc1d0b4643; no voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75
I20240501 14:02:27.704422 11118 raft_consensus.cc:3037] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.707165 11118 raft_consensus.cc:2726] T dfd4544926c04f7fa34419a8108e8d1e P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:27.710012 10928 leader_election.cc:302] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters: 5e09841056a14c29be2546cc1d0b4643
I20240501 14:02:27.710600 11159 raft_consensus.cc:2781] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:27.710708 11159 raft_consensus.cc:482] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:27.710791 11159 raft_consensus.cc:3037] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.717012 11159 raft_consensus.cc:504] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.718531 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552cf2dae53e4e81aad0bf5453eef53d" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643"
I20240501 14:02:27.718941 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552cf2dae53e4e81aad0bf5453eef53d" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48"
I20240501 14:02:27.719182 10857 raft_consensus.cc:3037] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:02:27.719401 10927 leader_election.cc:341] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:27.722329 10857 raft_consensus.cc:2454] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 1.
I20240501 14:02:27.722942 10928 leader_election.cc:302] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters: 5e09841056a14c29be2546cc1d0b4643
I20240501 14:02:27.723307 11159 leader_election.cc:288] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 election: Requested vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:27.723385 11116 raft_consensus.cc:2781] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:27.723574 11116 raft_consensus.cc:686] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 LEADER]: Becoming Leader. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Running, Role: LEADER
I20240501 14:02:27.723856 11116 consensus_queue.cc:237] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.726997 11109 tablet_bootstrap.cc:492] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.727197 11109 ts_tablet_manager.cc:1367] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.002s
I20240501 14:02:27.728018 11109 raft_consensus.cc:348] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.728358 11109 raft_consensus.cc:374] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.728485 11109 raft_consensus.cc:729] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.728781 11109 consensus_queue.cc:260] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.729420 11109 ts_tablet_manager.cc:1398] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:27.730791 11109 tablet_bootstrap.cc:492] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643: Bootstrap starting.
I20240501 14:02:27.732460 11109 tablet_bootstrap.cc:654] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:27.739358 11118 raft_consensus.cc:482] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.739560 11118 raft_consensus.cc:504] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.740307 11118 leader_election.cc:288] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.741506 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf192a1272427d899909c6bad20024" candidate_uuid: "5e09841056a14c29be2546cc1d0b4643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48" is_pre_election: true
I20240501 14:02:27.741523 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf192a1272427d899909c6bad20024" candidate_uuid: "5e09841056a14c29be2546cc1d0b4643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" is_pre_election: true
I20240501 14:02:27.741781 10992 raft_consensus.cc:2454] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e09841056a14c29be2546cc1d0b4643 in term 0.
I20240501 14:02:27.741784 10857 raft_consensus.cc:2454] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e09841056a14c29be2546cc1d0b4643 in term 0.
I20240501 14:02:27.742579 10656 leader_election.cc:302] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e09841056a14c29be2546cc1d0b4643, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:27.743126 11118 raft_consensus.cc:2781] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:27.743261 11118 raft_consensus.cc:482] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:27.743355 11118 raft_consensus.cc:3037] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.746179 11109 tablet_bootstrap.cc:492] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643: No bootstrap required, opened a new log
I20240501 14:02:27.746140 11118 raft_consensus.cc:504] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.746337 11109 ts_tablet_manager.cc:1367] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20240501 14:02:27.746987 11118 leader_election.cc:288] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 election: Requested vote from peers 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.747205 11109 raft_consensus.cc:348] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.747398 11109 raft_consensus.cc:374] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:27.747502 11109 raft_consensus.cc:729] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Initialized, Role: FOLLOWER
I20240501 14:02:27.747742 11109 consensus_queue.cc:260] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.748303 11109 ts_tablet_manager.cc:1398] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:27.748304 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf192a1272427d899909c6bad20024" candidate_uuid: "5e09841056a14c29be2546cc1d0b4643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75"
I20240501 14:02:27.748510 10992 raft_consensus.cc:3037] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.749450 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf192a1272427d899909c6bad20024" candidate_uuid: "5e09841056a14c29be2546cc1d0b4643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48"
I20240501 14:02:27.749682 10857 raft_consensus.cc:3037] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.754860 10992 raft_consensus.cc:2454] T 2ddf192a1272427d899909c6bad20024 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e09841056a14c29be2546cc1d0b4643 in term 1.
I20240501 14:02:27.756881 10656 leader_election.cc:302] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e09841056a14c29be2546cc1d0b4643, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:27.757628 10857 raft_consensus.cc:2454] T 2ddf192a1272427d899909c6bad20024 P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e09841056a14c29be2546cc1d0b4643 in term 1.
I20240501 14:02:27.759305 11118 raft_consensus.cc:2781] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:27.759539 11118 raft_consensus.cc:686] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [term 1 LEADER]: Becoming Leader. State: Replica: 5e09841056a14c29be2546cc1d0b4643, State: Running, Role: LEADER
I20240501 14:02:27.760062 11118 consensus_queue.cc:237] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.767413 11108 raft_consensus.cc:482] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.767674 11108 raft_consensus.cc:504] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.768389 11108 leader_election.cc:288] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.769186 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6038c451a846c6b6541c0ac115ca12" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643" is_pre_election: true
I20240501 14:02:27.769488 10720 raft_consensus.cc:2454] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 0.
I20240501 14:02:27.769982 10793 leader_election.cc:302] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, 5e09841056a14c29be2546cc1d0b4643; no voters:
I20240501 14:02:27.770787 11108 raft_consensus.cc:2781] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:27.770882 11108 raft_consensus.cc:482] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:27.770983 11108 raft_consensus.cc:3037] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.770994 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6038c451a846c6b6541c0ac115ca12" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" is_pre_election: true
I20240501 14:02:27.771270 10992 raft_consensus.cc:2454] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 0.
I20240501 14:02:27.773962 11108 raft_consensus.cc:504] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.775137 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6038c451a846c6b6541c0ac115ca12" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643"
I20240501 14:02:27.775414 10720 raft_consensus.cc:3037] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.780759 10720 raft_consensus.cc:2454] T fd6038c451a846c6b6541c0ac115ca12 P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 1.
I20240501 14:02:27.781327 10793 leader_election.cc:302] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, 5e09841056a14c29be2546cc1d0b4643; no voters:
I20240501 14:02:27.781677 11140 raft_consensus.cc:2781] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:27.784583 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6038c451a846c6b6541c0ac115ca12" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75"
I20240501 14:02:27.784868 10992 raft_consensus.cc:3037] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.786433 11108 leader_election.cc:288] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 election: Requested vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.787600 10992 raft_consensus.cc:2454] T fd6038c451a846c6b6541c0ac115ca12 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 1.
I20240501 14:02:27.800396 11140 raft_consensus.cc:686] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [term 1 LEADER]: Becoming Leader. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Running, Role: LEADER
I20240501 14:02:27.800874 11140 consensus_queue.cc:237] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:27.828208 11140 raft_consensus.cc:482] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.828480 11140 raft_consensus.cc:504] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.829403 11140 leader_election.cc:288] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.830799 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543d48b213be415c9cd17bbf67e0bb5d" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643" is_pre_election: true
I20240501 14:02:27.831068 10720 raft_consensus.cc:2454] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 0.
I20240501 14:02:27.831487 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543d48b213be415c9cd17bbf67e0bb5d" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" is_pre_election: true
I20240501 14:02:27.831713 10992 raft_consensus.cc:2454] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 0.
I20240501 14:02:27.832320 10794 leader_election.cc:302] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:27.832623 11168 raft_consensus.cc:482] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:27.832788 11168 raft_consensus.cc:504] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.833601 11168 leader_election.cc:288] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.833806 11168 raft_consensus.cc:2781] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:27.833923 11168 raft_consensus.cc:482] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:27.834018 11168 raft_consensus.cc:3037] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.837728 11168 raft_consensus.cc:504] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.838459 11168 leader_election.cc:288] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 election: Requested vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:27.840691 10719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543d48b213be415c9cd17bbf67e0bb5d" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643"
I20240501 14:02:27.841027 10719 raft_consensus.cc:3037] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.841037 10991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543d48b213be415c9cd17bbf67e0bb5d" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75"
I20240501 14:02:27.841351 10991 raft_consensus.cc:3037] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.844102 10719 raft_consensus.cc:2454] T 543d48b213be415c9cd17bbf67e0bb5d P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 1.
I20240501 14:02:27.844110 10991 raft_consensus.cc:2454] T 543d48b213be415c9cd17bbf67e0bb5d P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 1.
I20240501 14:02:27.845221 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ae9258fe314b9b9271a8f25a085be2" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643" is_pre_election: true
I20240501 14:02:27.845508 10720 raft_consensus.cc:2454] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 0.
I20240501 14:02:27.844899 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ae9258fe314b9b9271a8f25a085be2" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" is_pre_election: true
I20240501 14:02:27.845916 10992 raft_consensus.cc:2454] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 0.
I20240501 14:02:27.846532 10794 leader_election.cc:302] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:27.846961 10794 leader_election.cc:302] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:27.847283 11108 raft_consensus.cc:2781] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:27.847440 11108 raft_consensus.cc:686] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [term 1 LEADER]: Becoming Leader. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Running, Role: LEADER
I20240501 14:02:27.847739 11108 consensus_queue.cc:237] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.849651 11140 raft_consensus.cc:2781] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:27.849757 11140 raft_consensus.cc:482] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:27.849859 11140 raft_consensus.cc:3037] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.852522 11140 raft_consensus.cc:504] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.853838 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ae9258fe314b9b9271a8f25a085be2" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643"
I20240501 14:02:27.854061 10720 raft_consensus.cc:3037] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.856690 10720 raft_consensus.cc:2454] T 37ae9258fe314b9b9271a8f25a085be2 P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 1.
I20240501 14:02:27.857369 10793 leader_election.cc:302] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, 5e09841056a14c29be2546cc1d0b4643; no voters:
I20240501 14:02:27.858045 11108 raft_consensus.cc:2781] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:27.858203 11108 raft_consensus.cc:686] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [term 1 LEADER]: Becoming Leader. State: Replica: 156c3e56589d4969b853b3b512301a48, State: Running, Role: LEADER
I20240501 14:02:27.858506 11108 consensus_queue.cc:237] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:27.862629 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ae9258fe314b9b9271a8f25a085be2" candidate_uuid: "156c3e56589d4969b853b3b512301a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75"
I20240501 14:02:27.862875 10992 raft_consensus.cc:3037] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:27.865975 10992 raft_consensus.cc:2454] T 37ae9258fe314b9b9271a8f25a085be2 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156c3e56589d4969b853b3b512301a48 in term 1.
I20240501 14:02:27.866755 11140 leader_election.cc:288] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [CANDIDATE]: Term 1 election: Requested vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
14:02:27.906 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_381d9eb21d47411f8927a03d0f96a650 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20240501 14:02:27.908829 10451 catalog_manager.cc:2141] Already present: failed to create HMS catalog entry for table [id=74ce60b7803b4e7f908da24fa3d39ae6]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_381d9eb21d47411f8927a03d0f96a650 already exists)
I20240501 14:02:27.909412 10451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:59138 (request call id 11) took 1305 ms. Trace:
I20240501 14:02:27.909567 10451 rpcz_store.cc:276] 0501 14:02:26.604108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:02:26.604177 (+ 69us) service_pool.cc:225] Handling call
0501 14:02:26.628116 (+ 23939us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 14:02:26.628540 (+ 424us) catalog_manager.cc:2118] Created new table and tablet info
0501 14:02:27.909353 (+1280813us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":913746,"Hive Metastore.run_cpu_time_us":1027,"Hive Metastore.run_wall_time_us":366204}
I20240501 14:02:27.919781 10451 catalog_manager.cc:5484] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131). New cstate: current_term: 1 leader_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } health_report { overall_health: UNKNOWN } } }
I20240501 14:02:27.922637 10451 catalog_manager.cc:5484] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131). New cstate: current_term: 1 leader_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } health_report { overall_health: HEALTHY } } }
F20240501 14:02:27.928818 10634 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_498fa9129a934033b9b363449b003bef
*** Check failure stack trace: ***
@ 0x7f4e5407fdcd google::LogMessage::Fail() at ??:0
@ 0x7f4e54083b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4e5407f7cc google::LogMessage::Flush() at ??:0
I20240501 14:02:27.934159 10451 catalog_manager.cc:5484] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 156c3e56589d4969b853b3b512301a48 (127.9.224.130). New cstate: current_term: 1 leader_uuid: "156c3e56589d4969b853b3b512301a48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } health_report { overall_health: UNKNOWN } } }
@ 0x7f4e5407fa11 google::LogMessage::~LogMessage() at ??:0
I20240501 14:02:27.935025 10451 catalog_manager.cc:5484] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 156c3e56589d4969b853b3b512301a48 (127.9.224.130). New cstate: current_term: 1 leader_uuid: "156c3e56589d4969b853b3b512301a48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } health_report { overall_health: HEALTHY } } }
I20240501 14:02:27.935459 10451 catalog_manager.cc:5484] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 156c3e56589d4969b853b3b512301a48 (127.9.224.130). New cstate: current_term: 1 leader_uuid: "156c3e56589d4969b853b3b512301a48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } health_report { overall_health: UNKNOWN } } }
@ 0x7f4e6a29457e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f4e6a293261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240501 14:02:27.948518 10451 catalog_manager.cc:5484] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129). New cstate: current_term: 1 leader_uuid: "5e09841056a14c29be2546cc1d0b4643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } health_report { overall_health: UNKNOWN } } }
@ 0x7f4e6a29a2bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f4e6a29a139 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f4e56d6f01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4e5a0f46db start_thread at ??:0
@ 0x7f4e52f3271f clone at ??:0
I20240501 14:02:28.087839 11116 raft_consensus.cc:482] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:28.088101 11116 raft_consensus.cc:504] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:28.089578 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ff9d28dcf24f24935f5c739316ff70" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643" is_pre_election: true
I20240501 14:02:28.089622 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ff9d28dcf24f24935f5c739316ff70" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48" is_pre_election: true
I20240501 14:02:28.089895 10720 raft_consensus.cc:2454] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 0.
I20240501 14:02:28.090001 10857 raft_consensus.cc:2454] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 0.
I20240501 14:02:28.090534 10928 leader_election.cc:302] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:28.091212 11158 raft_consensus.cc:2781] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:28.091311 11158 raft_consensus.cc:482] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:28.091401 11158 raft_consensus.cc:3037] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:28.094425 11158 raft_consensus.cc:504] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:28.095858 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ff9d28dcf24f24935f5c739316ff70" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643"
I20240501 14:02:28.096118 10720 raft_consensus.cc:3037] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:28.096074 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ff9d28dcf24f24935f5c739316ff70" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48"
I20240501 14:02:28.096282 10857 raft_consensus.cc:3037] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:28.088934 11116 leader_election.cc:288] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:28.099735 10857 raft_consensus.cc:2454] T 56ff9d28dcf24f24935f5c739316ff70 P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 1.
I20240501 14:02:28.099637 10720 raft_consensus.cc:2454] T 56ff9d28dcf24f24935f5c739316ff70 P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 1.
I20240501 14:02:28.100497 10928 leader_election.cc:302] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:28.100829 11159 raft_consensus.cc:482] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:28.101004 11159 raft_consensus.cc:504] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:28.101189 11116 raft_consensus.cc:2781] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:28.101552 11158 leader_election.cc:288] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 election: Requested vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:28.101763 11159 leader_election.cc:288] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:28.101989 11116 raft_consensus.cc:686] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 LEADER]: Becoming Leader. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Running, Role: LEADER
I20240501 14:02:28.102277 11116 consensus_queue.cc:237] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } }
I20240501 14:02:28.105798 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b747db1c8b2b4921b3a1d23012f8b3a8" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48" is_pre_election: true
I20240501 14:02:28.106269 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b747db1c8b2b4921b3a1d23012f8b3a8" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643" is_pre_election: true
I20240501 14:02:28.106601 10857 raft_consensus.cc:2454] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 0.
I20240501 14:02:28.107157 10928 leader_election.cc:302] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:28.107661 10720 raft_consensus.cc:2454] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 0.
I20240501 14:02:28.108541 11185 raft_consensus.cc:2781] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:28.108703 11185 raft_consensus.cc:482] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:28.108866 11185 raft_consensus.cc:3037] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:28.112087 11185 raft_consensus.cc:504] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:28.113405 10720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b747db1c8b2b4921b3a1d23012f8b3a8" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e09841056a14c29be2546cc1d0b4643"
I20240501 14:02:28.113634 10720 raft_consensus.cc:3037] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:28.116429 10720 raft_consensus.cc:2454] T b747db1c8b2b4921b3a1d23012f8b3a8 P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 1.
I20240501 14:02:28.116874 11185 leader_election.cc:288] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 election: Requested vote from peers 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:28.117216 10927 leader_election.cc:302] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e09841056a14c29be2546cc1d0b4643, c1d9952ad6a24cecbcde3d02a59c9a75; no voters:
I20240501 14:02:28.117537 11185 raft_consensus.cc:2781] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:28.117523 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b747db1c8b2b4921b3a1d23012f8b3a8" candidate_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48"
I20240501 14:02:28.117722 11185 raft_consensus.cc:686] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [term 1 LEADER]: Becoming Leader. State: Replica: c1d9952ad6a24cecbcde3d02a59c9a75, State: Running, Role: LEADER
I20240501 14:02:28.117760 10857 raft_consensus.cc:3037] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:28.117982 11185 consensus_queue.cc:237] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:28.120502 10857 raft_consensus.cc:2454] T b747db1c8b2b4921b3a1d23012f8b3a8 P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in term 1.
W20240501 14:02:28.132499 11152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:28.132863 11152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:28.133069 11152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:28.141961 11152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:28.142100 11152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:28.142172 11152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:28.142246 11152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:28.151935 11152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.224.188
--webserver_port=35625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:28.153440 11152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20240501 14:02:28.154059 10925 connection.cc:535] client connection to 127.9.224.189:42871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:28.154691 11044 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.224.189:42871)
W20240501 14:02:28.155261 11044 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.224.189:42871)
W20240501 14:02:28.155783 11044 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.224.189:42871)
W20240501 14:02:28.156415 10374 connection.cc:535] server connection from 127.0.0.1:55522 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:28.157466 11152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:28.157701 11044 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.224.189:42871)
W20240501 14:02:28.158720 11157 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:28.154936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.224.189:42871 (local address 127.9.224.131:44019)
0501 14:02:28.155592 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:28.155613 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:28.155742 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:28.157647 (+ 1905us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":524}
W20240501 14:02:28.172724 11195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:28.173163 11194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:28.173540 11197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:28.174640 11152 server_base.cc:1034] running on GCE node
I20240501 14:02:28.175797 11152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:28.177611 11152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:28.179138 11152 hybrid_clock.cc:648] HybridClock initialized: now 1714572148179025 us; error 69 us; skew 500 ppm
I20240501 14:02:28.179852 11152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:28.183630 11152 webserver.cc:460] Webserver started at http://127.9.224.188:35625/ using document root <none> and password file <none>
I20240501 14:02:28.184584 11152 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:28.184818 11152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:28.192865 11152 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:02:28.196190 11203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:28.197984 11152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 14:02:28.198194 11152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
uuid: "4163d103537745edb461e4dbdda724e6"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-gtmz"
I20240501 14:02:28.199184 11152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:28.220741 11152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:28.221741 11152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:28.222191 11152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:28.240911 11152 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.188:38257
I20240501 14:02:28.240924 11254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.188:38257 every 8 connection(s)
I20240501 14:02:28.242427 11152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
I20240501 14:02:28.249790 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 11152
I20240501 14:02:28.250069 11255 sys_catalog.cc:263] Verifying existing consensus state
I20240501 14:02:28.253504 11166 raft_consensus.cc:482] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:28.253702 11166 raft_consensus.cc:504] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 } } peers { permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 } } peers { permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 } }
I20240501 14:02:28.254427 11166 leader_election.cc:288] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371), 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:28.255592 11255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap starting.
I20240501 14:02:28.255522 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552cf2dae53e4e81aad0bf5453eef53d" candidate_uuid: "5e09841056a14c29be2546cc1d0b4643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" is_pre_election: true
I20240501 14:02:28.256091 10857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552cf2dae53e4e81aad0bf5453eef53d" candidate_uuid: "5e09841056a14c29be2546cc1d0b4643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156c3e56589d4969b853b3b512301a48" is_pre_election: true
I20240501 14:02:28.256405 10857 raft_consensus.cc:2379] T 552cf2dae53e4e81aad0bf5453eef53d P 156c3e56589d4969b853b3b512301a48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e09841056a14c29be2546cc1d0b4643 in current term 1: Already voted for candidate c1d9952ad6a24cecbcde3d02a59c9a75 in this term.
I20240501 14:02:28.257179 10656 leader_election.cc:302] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e09841056a14c29be2546cc1d0b4643; no voters: 156c3e56589d4969b853b3b512301a48, c1d9952ad6a24cecbcde3d02a59c9a75
I20240501 14:02:28.257622 11166 raft_consensus.cc:3037] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:28.260749 11166 raft_consensus.cc:2726] T 552cf2dae53e4e81aad0bf5453eef53d P 5e09841056a14c29be2546cc1d0b4643 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:28.266232 11255 log.cc:824] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:28.275851 11255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:28.276417 11255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap complete.
I20240501 14:02:28.284315 10765 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:28.285358 11255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:28.286525 11255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4163d103537745edb461e4dbdda724e6, State: Initialized, Role: FOLLOWER
I20240501 14:02:28.287343 11255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:28.288305 11261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:28.288657 11261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:28.289537 11255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:28.328411 11275 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:28.328560 11275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: loading cluster ID for follower catalog manager: success
I20240501 14:02:28.333245 11275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: acquiring CA information for follower catalog manager: success
I20240501 14:02:28.336094 11275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:28.344657 11108 consensus_queue.cc:1035] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.362399 11037 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:28.362663 11037 heartbeater.cc:503] Master 127.9.224.188:38257 was elected leader, sending a full tablet report...
I20240501 14:02:28.369362 11140 consensus_queue.cc:1035] T dfd4544926c04f7fa34419a8108e8d1e P 156c3e56589d4969b853b3b512301a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.392017 11219 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" instance_seqno: 1714572146262436) as {username='slave'} at 127.9.224.131:49715; Asking this server to re-register.
I20240501 14:02:28.393517 11037 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:28.365865 11152 master_runner.cc:186] Error getting master registration for 127.9.224.189:42871: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111),
I20240501 14:02:28.393746 11152 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:28.396023 11219 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:28.398730 11219 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" instance_seqno: 1714572145337112) as {username='slave'} at 127.9.224.129:33291; Asking this server to re-register.
I20240501 14:02:28.399739 10765 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:28.403229 11220 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:28.403674 10903 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:28.405786 11220 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "156c3e56589d4969b853b3b512301a48" instance_seqno: 1714572145819154) as {username='slave'} at 127.9.224.130:41203; Asking this server to re-register.
I20240501 14:02:28.406491 10903 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:28.407951 11220 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:28.428390 11185 consensus_queue.cc:1035] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.434890 11158 consensus_queue.cc:1035] T 0496a42cfb744b25ad294d426ded937f P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.464412 11290 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e048fbcc4e544adea37b6bd9e5ac8543)
I20240501 14:02:28.464612 11290 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:28.465332 11290 leader_election.cc:288] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
W20240501 14:02:28.466848 10372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240501 14:02:28.468396 10372 leader_election.cc:334] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:28.474092 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6" is_pre_election: true
I20240501 14:02:28.474741 11230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47f54722b654655bfc871851b93abbc in term 1.
I20240501 14:02:28.475327 10375 leader_election.cc:302] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters: e048fbcc4e544adea37b6bd9e5ac8543
I20240501 14:02:28.475634 11290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:02:28.475759 11290 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 2 FOLLOWER]: Starting leader election (detected failure of leader e048fbcc4e544adea37b6bd9e5ac8543)
I20240501 14:02:28.475857 11290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:02:28.477913 11290 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:28.478556 11290 leader_election.cc:288] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 3 election: Requested vote from peers e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
I20240501 14:02:28.479179 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6"
I20240501 14:02:28.479414 11230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 1 FOLLOWER]: Advancing to term 3
W20240501 14:02:28.480221 10372 leader_election.cc:334] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:28.485754 11230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47f54722b654655bfc871851b93abbc in term 3.
I20240501 14:02:28.486462 10375 leader_election.cc:302] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters: e048fbcc4e544adea37b6bd9e5ac8543
I20240501 14:02:28.486902 11290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:02:28.487396 11290 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 LEADER]: Becoming Leader. State: Replica: e47f54722b654655bfc871851b93abbc, State: Running, Role: LEADER
I20240501 14:02:28.487973 11290 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:28.489866 11294 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47f54722b654655bfc871851b93abbc. Latest consensus state: current_term: 3 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:28.490232 11294 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:28.491142 11295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:28.494678 11295 catalog_manager.cc:669] Loaded metadata for table default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7]
I20240501 14:02:28.495306 11295 catalog_manager.cc:669] Loaded metadata for table default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd]
I20240501 14:02:28.495777 11295 catalog_manager.cc:669] Loaded metadata for table default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5]
I20240501 14:02:28.496237 11295 catalog_manager.cc:669] Loaded metadata for table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]
I20240501 14:02:28.496665 11295 catalog_manager.cc:669] Loaded metadata for table default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3]
I20240501 14:02:28.497138 11295 catalog_manager.cc:669] Loaded metadata for table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]
I20240501 14:02:28.497567 11295 catalog_manager.cc:669] Loaded metadata for table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]
I20240501 14:02:28.499070 11295 catalog_manager.cc:669] Loaded metadata for table default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323]
I20240501 14:02:28.499511 11295 catalog_manager.cc:669] Loaded metadata for table default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8]
I20240501 14:02:28.502249 11295 tablet_loader.cc:96] loaded metadata for tablet 0496a42cfb744b25ad294d426ded937f (table default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7])
I20240501 14:02:28.502735 11295 tablet_loader.cc:96] loaded metadata for tablet 2ddf192a1272427d899909c6bad20024 (table default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3])
I20240501 14:02:28.503167 11295 tablet_loader.cc:96] loaded metadata for tablet 37ae9258fe314b9b9271a8f25a085be2 (table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f])
I20240501 14:02:28.503548 11295 tablet_loader.cc:96] loaded metadata for tablet 543d48b213be415c9cd17bbf67e0bb5d (table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb])
I20240501 14:02:28.503913 11295 tablet_loader.cc:96] loaded metadata for tablet 552cf2dae53e4e81aad0bf5453eef53d (table default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5])
I20240501 14:02:28.504289 11295 tablet_loader.cc:96] loaded metadata for tablet 56ff9d28dcf24f24935f5c739316ff70 (table default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8])
I20240501 14:02:28.505209 11295 tablet_loader.cc:96] loaded metadata for tablet b747db1c8b2b4921b3a1d23012f8b3a8 (table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043])
I20240501 14:02:28.505846 11295 tablet_loader.cc:96] loaded metadata for tablet dfd4544926c04f7fa34419a8108e8d1e (table default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323])
I20240501 14:02:28.506242 11295 tablet_loader.cc:96] loaded metadata for tablet fd6038c451a846c6b6541c0ac115ca12 (table default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd])
I20240501 14:02:28.506582 11295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:28.508141 11295 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:28.508246 11295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:28.510172 11295 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:28.511533 11295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Loaded TSK: 0
I20240501 14:02:28.512734 11295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:28.514159 11295 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 14:02:28.515416 11295 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240501 14:02:28.527761 11158 consensus_queue.cc:1035] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.535827 11158 consensus_queue.cc:1035] T 552cf2dae53e4e81aad0bf5453eef53d P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.538173 11166 consensus_queue.cc:1035] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.541102 10904 heartbeater.cc:503] Master 127.9.224.190:34475 was elected leader, sending a full tablet report...
I20240501 14:02:28.542922 10388 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55550:
name: "default.Table_557e6f54359d445eab8bcfb616e974ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:28.544171 10385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55550:
name: "default.Table_1a9db37a35094b309731f45b45ef8113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:28.544947 10383 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55550:
name: "default.Table_a67258cb839442b69e96ce75fc4c281b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:28.545493 10382 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55550:
table { table_name: "default.Table_d67cd2442dc24ffc8d4dc86e19a50315" } modify_external_catalogs: true
I20240501 14:02:28.545766 10382 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55550:
table { table_name: "default.Table_d67cd2442dc24ffc8d4dc86e19a50315" } modify_external_catalogs: true
I20240501 14:02:28.546494 10381 master_service.cc:946] ReplaceTablet: received request to replace tablet fd6038c451a846c6b6541c0ac115ca12 from {username='slave'} at 127.0.0.1:55550
I20240501 14:02:28.546713 10380 master_service.cc:946] ReplaceTablet: received request to replace tablet b747db1c8b2b4921b3a1d23012f8b3a8 from {username='slave'} at 127.0.0.1:55550
I20240501 14:02:28.547730 10379 master_service.cc:946] ReplaceTablet: received request to replace tablet 0496a42cfb744b25ad294d426ded937f from {username='slave'} at 127.0.0.1:55550
I20240501 14:02:28.549283 10384 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55550:
table { table_name: "default.Table_740f0f0aef9c46919f5658f6d2be21e3" } modify_external_catalogs: true
I20240501 14:02:28.549561 10384 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55550:
table { table_name: "default.Table_740f0f0aef9c46919f5658f6d2be21e3" } modify_external_catalogs: true
I20240501 14:02:28.549880 10387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55550:
name: "default.Table_f616bd9bd35f4659896ecc727f466014"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:28.551028 10386 master_service.cc:946] ReplaceTablet: received request to replace tablet dfd4544926c04f7fa34419a8108e8d1e from {username='slave'} at 127.0.0.1:55550
W20240501 14:02:28.554566 10372 consensus_peers.cc:485] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc -> Peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Couldn't send request to peer e048fbcc4e544adea37b6bd9e5ac8543. Status: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20240501 14:02:28.560132 10585 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_498fa9129a934033b9b363449b003bef
*** Check failure stack trace: ***
I20240501 14:02:28.561326 11230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Refusing update from remote peer e47f54722b654655bfc871851b93abbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240501 14:02:28.562356 11294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7fb77a091dcd google::LogMessage::Fail() at ??:0
@ 0x7fb77a095b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb77a0917cc google::LogMessage::Flush() at ??:0
@ 0x7fb77a091a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fb7902a657e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240501 14:02:28.584059 11297 consensus_queue.cc:1035] T 2ddf192a1272427d899909c6bad20024 P 5e09841056a14c29be2546cc1d0b4643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.586179 11038 heartbeater.cc:503] Master 127.9.224.190:34475 was elected leader, sending a full tablet report...
@ 0x7fb7902a5261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fb7902ac2bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fb7902ac139 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb77cd8101e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb7801066db start_thread at ??:0
@ 0x7fb778f4471f clone at ??:0
I20240501 14:02:28.652902 11261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47f54722b654655bfc871851b93abbc. Latest consensus state: current_term: 3 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:28.653281 11261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:28.655352 11305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:28.655719 11305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:28.675159 11301 mvcc.cc:205] Tried to move back new op lower bound from 7022887518025437184 to 7022887512299618304. Current Snapshot: MvccSnapshot[applied={T|T < 7022887518025437184 or (T in {7022887518025437184})}]
I20240501 14:02:28.704277 11140 consensus_queue.cc:1035] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.709906 11108 consensus_queue.cc:1035] T 543d48b213be415c9cd17bbf67e0bb5d P 156c3e56589d4969b853b3b512301a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.760320 11108 consensus_queue.cc:1035] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.768671 11140 consensus_queue.cc:1035] T 37ae9258fe314b9b9271a8f25a085be2 P 156c3e56589d4969b853b3b512301a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:28.830438 11207 connection.cc:535] server connection from 127.0.0.1:42852 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:28.831482 11157 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:28.829156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.224.190:34475 (local address 127.9.224.131:43793)
0501 14:02:28.829411 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:28.829432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:28.829555 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:28.831295 (+ 1740us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135}
W20240501 14:02:28.832154 11285 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:28.829305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.224.190:34475 (local address 127.9.224.130:43307)
0501 14:02:28.831223 (+ 1918us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:28.831298 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1811}
I20240501 14:02:28.883870 11168 consensus_queue.cc:1035] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" member_type: VOTER last_known_addr { host: "127.9.224.131" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:28.890342 11140 consensus_queue.cc:1035] T fd6038c451a846c6b6541c0ac115ca12 P 156c3e56589d4969b853b3b512301a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:29.067080 11305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e47f54722b654655bfc871851b93abbc)
I20240501 14:02:29.067373 11305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:29.069720 11305 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:29.070626 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:29.072054 11205 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:29.072579 11204 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:29.072734 11204 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:29.073220 11305 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 14:02:29.129508 11158 consensus_queue.cc:1035] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:29.136039 11334 consensus_queue.cc:1035] T 56ff9d28dcf24f24935f5c739316ff70 P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:29.303321 11158 consensus_queue.cc:1035] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" member_type: VOTER last_known_addr { host: "127.9.224.129" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:29.310073 11158 consensus_queue.cc:1035] T b747db1c8b2b4921b3a1d23012f8b3a8 P c1d9952ad6a24cecbcde3d02a59c9a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156c3e56589d4969b853b3b512301a48" member_type: VOTER last_known_addr { host: "127.9.224.130" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:29.603626 11346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e47f54722b654655bfc871851b93abbc)
I20240501 14:02:29.603809 11346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:29.604475 11346 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:29.606316 11205 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:29.606489 11204 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:29.606633 11204 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:29.606878 11346 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 14:02:30.387260 11351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e47f54722b654655bfc871851b93abbc)
I20240501 14:02:30.387470 11351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:30.388121 11351 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:30.390467 11205 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:30.390646 11204 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:30.390770 11204 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:30.391124 11351 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 14:02:31.503302 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.190:34475
--webserver_interface=127.9.224.190
--webserver_port=35165
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 14:02:31.839773 11363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:31.840186 11363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:31.840325 11363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:31.849666 11363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:31.849785 11363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:31.849841 11363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:31.849897 11363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:31.859675 11363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.190:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.224.190
--webserver_port=35165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:31.861007 11363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:31.862812 11363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:31.873847 11374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:31.873895 11372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:31.874459 11371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:31.874879 11363 server_base.cc:1034] running on GCE node
I20240501 14:02:31.875653 11363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:31.877143 11363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:31.878369 11363 hybrid_clock.cc:648] HybridClock initialized: now 1714572151878297 us; error 56 us; skew 500 ppm
I20240501 14:02:31.878713 11363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:31.881052 11363 webserver.cc:460] Webserver started at http://127.9.224.190:35165/ using document root <none> and password file <none>
I20240501 14:02:31.881810 11363 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:31.881953 11363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:31.887385 11363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:02:31.890408 11380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:31.891856 11363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:31.892031 11363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
uuid: "e47f54722b654655bfc871851b93abbc"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-gtmz"
I20240501 14:02:31.892890 11363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:31.908453 11363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:31.909446 11363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:31.909727 11363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:31.928196 11363 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.190:34475
I20240501 14:02:31.928231 11431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.190:34475 every 8 connection(s)
I20240501 14:02:31.929742 11363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
I20240501 14:02:31.930560 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 11363
I20240501 14:02:31.937081 11432 sys_catalog.cc:263] Verifying existing consensus state
I20240501 14:02:31.941277 11432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap starting.
I20240501 14:02:31.951936 11432 log.cc:824] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:31.979616 11432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=37 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=22 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20240501 14:02:31.980274 11432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap complete.
I20240501 14:02:31.987123 11432 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:31.990962 11432 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47f54722b654655bfc871851b93abbc, State: Initialized, Role: FOLLOWER
I20240501 14:02:31.991750 11432 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:31.992636 11437 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:31.992925 11437 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:31.993722 11432 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:32.005740 11451 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:32.005954 11451 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: loading cluster ID for follower catalog manager: success
I20240501 14:02:32.010368 11451 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: acquiring CA information for follower catalog manager: success
I20240501 14:02:32.013842 11451 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:32.037350 11460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e47f54722b654655bfc871851b93abbc)
I20240501 14:02:32.038177 11460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:32.043560 10904 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:32.048331 11397 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "156c3e56589d4969b853b3b512301a48" instance_seqno: 1714572145819154) as {username='slave'} at 127.9.224.130:55461; Asking this server to re-register.
I20240501 14:02:32.050012 10904 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:32.053231 11396 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:32.054376 11460 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
I20240501 14:02:32.063112 11407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4163d103537745edb461e4dbdda724e6" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "e47f54722b654655bfc871851b93abbc" is_pre_election: true
I20240501 14:02:32.064008 11407 raft_consensus.cc:2396] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4163d103537745edb461e4dbdda724e6 for term 4 because replica has last-logged OpId of term: 3 index: 37, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 35.
I20240501 14:02:32.065724 10766 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:32.068529 11396 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" instance_seqno: 1714572145337112) as {username='slave'} at 127.9.224.129:39525; Asking this server to re-register.
I20240501 14:02:32.069437 10766 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:32.071427 11396 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
W20240501 14:02:32.073892 11204 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:32.074054 11204 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:32.074335 11460 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 14:02:32.070506 11363 master_runner.cc:186] Error getting master registration for 127.9.224.189:42871: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111),
I20240501 14:02:32.078289 11363 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:32.099154 11038 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:32.099362 11038 heartbeater.cc:503] Master 127.9.224.190:34475 was elected leader, sending a full tablet report...
I20240501 14:02:32.101167 11396 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" instance_seqno: 1714572146262436) as {username='slave'} at 127.9.224.131:43519; Asking this server to re-register.
I20240501 14:02:32.101821 11038 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:32.103314 11396 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:32.497633 11469 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:32.497936 11469 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:32.499737 11469 leader_election.cc:288] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
W20240501 14:02:32.500638 11381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:32.502106 11381 leader_election.cc:334] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:32.506129 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6" is_pre_election: true
I20240501 14:02:32.506422 11230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47f54722b654655bfc871851b93abbc in term 3.
I20240501 14:02:32.506949 11384 leader_election.cc:302] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters: e048fbcc4e544adea37b6bd9e5ac8543
I20240501 14:02:32.507373 11469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 14:02:32.507588 11469 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:32.507706 11469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:02:32.511574 11469 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:32.512312 11469 leader_election.cc:288] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 4 election: Requested vote from peers e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
I20240501 14:02:32.513010 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6"
I20240501 14:02:32.513306 11230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 3 FOLLOWER]: Advancing to term 4
W20240501 14:02:32.513695 11381 leader_election.cc:334] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:32.515341 11230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47f54722b654655bfc871851b93abbc in term 4.
I20240501 14:02:32.515790 11384 leader_election.cc:302] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters: e048fbcc4e544adea37b6bd9e5ac8543
I20240501 14:02:32.516116 11469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:02:32.516517 11469 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 LEADER]: Becoming Leader. State: Replica: e47f54722b654655bfc871851b93abbc, State: Running, Role: LEADER
I20240501 14:02:32.517030 11469 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.37, Last appended by leader: 37, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:32.520567 11472 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47f54722b654655bfc871851b93abbc. Latest consensus state: current_term: 4 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:32.520915 11472 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:32.906430 11230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Refusing update from remote peer e47f54722b654655bfc871851b93abbc: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 4 index: 38. (index mismatch)
I20240501 14:02:32.907423 11472 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:02:32.911397 11476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47f54722b654655bfc871851b93abbc. Latest consensus state: current_term: 4 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:32.911680 11476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:32.917337 11472 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:32.917698 11472 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:32.917981 11486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:32.918351 11486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:32.918242 11485 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:32.918607 11476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:32.918844 11476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:32.919072 11485 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: LEADER
W20240501 14:02:32.919292 11381 consensus_peers.cc:485] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc -> Peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Couldn't send request to peer e048fbcc4e544adea37b6bd9e5ac8543. Status: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:32.935215 11474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:32.938004 11474 catalog_manager.cc:669] Loaded metadata for table default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7]
I20240501 14:02:32.938596 11474 catalog_manager.cc:669] Loaded metadata for table default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd]
I20240501 14:02:32.939071 11474 catalog_manager.cc:669] Loaded metadata for table default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5]
I20240501 14:02:32.939529 11474 catalog_manager.cc:669] Loaded metadata for table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]
I20240501 14:02:32.939906 11474 catalog_manager.cc:669] Loaded metadata for table default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3]
I20240501 14:02:32.940232 11474 catalog_manager.cc:669] Loaded metadata for table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]
I20240501 14:02:32.940567 11474 catalog_manager.cc:669] Loaded metadata for table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]
I20240501 14:02:32.940919 11474 catalog_manager.cc:669] Loaded metadata for table default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323]
I20240501 14:02:32.941264 11474 catalog_manager.cc:669] Loaded metadata for table default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8]
I20240501 14:02:32.943266 11474 tablet_loader.cc:96] loaded metadata for tablet 2ddf192a1272427d899909c6bad20024 (table default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3])
I20240501 14:02:32.943593 11474 tablet_loader.cc:96] loaded metadata for tablet 37ae9258fe314b9b9271a8f25a085be2 (table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f])
I20240501 14:02:32.943925 11474 tablet_loader.cc:96] loaded metadata for tablet 543d48b213be415c9cd17bbf67e0bb5d (table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb])
I20240501 14:02:32.944218 11474 tablet_loader.cc:96] loaded metadata for tablet 552cf2dae53e4e81aad0bf5453eef53d (table default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5])
I20240501 14:02:32.944509 11474 tablet_loader.cc:96] loaded metadata for tablet 56ff9d28dcf24f24935f5c739316ff70 (table default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8])
I20240501 14:02:32.944751 11474 tablet_loader.cc:96] loaded metadata for tablet 645755771e714431ac3d429bfe6b01f0 (table default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd])
I20240501 14:02:32.945016 11474 tablet_loader.cc:96] loaded metadata for tablet 667ad182034b4dd2aaf55f1486d9318f (table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043])
I20240501 14:02:32.945559 11474 tablet_loader.cc:96] loaded metadata for tablet 81440735b20d43c2ae3d3913919a5bef (table default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7])
I20240501 14:02:32.946092 11474 tablet_loader.cc:96] loaded metadata for tablet dd9e4695e08d425aab0068cbcd673d8a (table default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323])
I20240501 14:02:32.946655 11474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:32.947674 11474 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:32.947749 11474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:32.949043 11474 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:32.949923 11474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Loaded TSK: 0
I20240501 14:02:32.950726 11474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:32.951610 11474 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 14:02:32.952442 11474 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240501 14:02:32.964669 11397 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36114:
table { table_name: "default.Table_e4a61f4822d448919e6decf48ba63a8d" } new_table_name: "default.Table_3bc74075f55d4af7a1e48e16115ae551" modify_external_catalogs: true
I20240501 14:02:32.964913 11396 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36114:
name: "default.Table_4801897b969f4fb8a4d46503057f4169"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:32.966370 11388 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36114:
name: "default.Table_a67258cb839442b69e96ce75fc4c281b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:32.966935 11395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36114:
table { table_name: "default.Table_6d8f3ef2579a49df82d7ae9d6fbb3e2b" } new_table_name: "default.Table_6cc7bd2d05ec418ba2a930d28e0257b0" modify_external_catalogs: true
I20240501 14:02:32.967082 11389 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36114:
table { table_name: "default.Table_d67cd2442dc24ffc8d4dc86e19a50315" } modify_external_catalogs: true
I20240501 14:02:32.967737 11389 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36114:
table { table_name: "default.Table_d67cd2442dc24ffc8d4dc86e19a50315" } modify_external_catalogs: true
I20240501 14:02:32.967392 11394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36114:
table { table_name: "default.Table_c9596eb80e944f919a8c85174a7080ca" } new_table_name: "default.Table_fa62350979764cedbfc470287d68ffe1" modify_external_catalogs: true
I20240501 14:02:32.967826 11392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36114:
table { table_name: "default.Table_cc9ce1d1031f42789771ae5eb4a6df43" } new_table_name: "default.Table_b8695d7d9f484f03995f17c0bbc61ee3" modify_external_catalogs: true
I20240501 14:02:32.968415 11393 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36114:
table { table_name: "default.Table_498fa9129a934033b9b363449b003bef" } new_table_name: "default.Table_ca677c5ce49e45d8ad3221682525d4c5" modify_external_catalogs: true
I20240501 14:02:32.968303 11391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36114:
name: "default.Table_f4862f63c8104085837aee11c33c2718"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:32.968187 11390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36114:
name: "default.Table_351f54255b01483daad53a39989a377c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20240501 14:02:32.971753 11450 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_498fa9129a934033b9b363449b003bef
*** Check failure stack trace: ***
@ 0x7f8895098dcd google::LogMessage::Fail() at ??:0
@ 0x7f889509cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f88950987cc google::LogMessage::Flush() at ??:0
@ 0x7f8895098a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f88ab2ad57e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f88ab2ac261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f88ab2b32bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f88ab2b3139 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f8897d8801e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f889b10d6db start_thread at ??:0
@ 0x7f8893f4b71f clone at ??:0
W20240501 14:02:33.161737 10926 connection.cc:535] client connection to 127.9.224.190:34475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:33.161801 10792 connection.cc:535] client connection to 127.9.224.190:34475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:33.162227 10654 connection.cc:535] client connection to 127.9.224.190:34475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:33.162760 11045 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.224.190:34475)
W20240501 14:02:33.162992 11045 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.224.190:34475)
W20240501 14:02:33.163460 11045 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.224.190:34475)
W20240501 14:02:33.163828 11045 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.224.190:34475)
I20240501 14:02:33.395011 11152 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:02:33.395196 11152 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:33.413583 11476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e47f54722b654655bfc871851b93abbc)
I20240501 14:02:33.413789 11476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:33.414494 11476 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:33.416626 11204 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:33.416842 11205 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:02:33.416954 11205 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:33.417245 11476 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 14:02:33.943316 11510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e47f54722b654655bfc871851b93abbc)
I20240501 14:02:33.943487 11510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:33.944229 11510 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:33.946590 11205 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:33.946750 11204 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:33.946883 11204 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:33.947110 11510 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 14:02:34.768510 11516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e47f54722b654655bfc871851b93abbc)
I20240501 14:02:34.768728 11516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:34.769446 11516 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:34.770241 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:02:34.771382 11205 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:34.771584 11204 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:34.771711 11204 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:34.771962 11516 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 14:02:34.901296 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.190:34475
--webserver_interface=127.9.224.190
--webserver_port=35165
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 14:02:35.228179 11520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:35.228511 11520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:35.228649 11520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:35.236397 11520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:35.236518 11520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:35.236562 11520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:35.236629 11520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:35.246017 11520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.190:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.224.190
--webserver_port=35165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:35.247426 11520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:35.249343 11520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:35.261305 11528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:35.261281 11525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:35.261847 11526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:35.266736 11520 server_base.cc:1034] running on GCE node
I20240501 14:02:35.267614 11520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:35.269153 11520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:35.270546 11520 hybrid_clock.cc:648] HybridClock initialized: now 1714572155270493 us; error 36 us; skew 500 ppm
I20240501 14:02:35.270931 11520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:35.279417 11520 webserver.cc:460] Webserver started at http://127.9.224.190:35165/ using document root <none> and password file <none>
I20240501 14:02:35.280222 11520 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:35.280351 11520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:35.286126 11520 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 14:02:35.289355 11534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:35.290928 11520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:02:35.291107 11520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
uuid: "e47f54722b654655bfc871851b93abbc"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-gtmz"
I20240501 14:02:35.292008 11520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:35.345736 11520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:35.346652 11520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:35.346940 11520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:35.365501 11520 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.190:34475
I20240501 14:02:35.365517 11586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.190:34475 every 8 connection(s)
I20240501 14:02:35.367063 11520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
I20240501 14:02:35.371696 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 11520
I20240501 14:02:35.374292 11587 sys_catalog.cc:263] Verifying existing consensus state
I20240501 14:02:35.378994 11587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap starting.
I20240501 14:02:35.393927 11587 log.cc:824] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:35.424415 11587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=28 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:35.424963 11587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap complete.
I20240501 14:02:35.431725 11587 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:35.432601 11587 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47f54722b654655bfc871851b93abbc, State: Initialized, Role: FOLLOWER
I20240501 14:02:35.433326 11587 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 4.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:35.434132 11591 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:35.434404 11591 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:35.435178 11587 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:35.448043 11605 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:35.448271 11605 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: loading cluster ID for follower catalog manager: success
I20240501 14:02:35.453135 11605 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: acquiring CA information for follower catalog manager: success
I20240501 14:02:35.456266 11605 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:35.457190 11520 master_runner.cc:186] Error getting master registration for 127.9.224.189:42871: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111),
I20240501 14:02:35.466864 11520 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:35.676713 10904 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:35.680459 11038 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:35.680290 11551 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "156c3e56589d4969b853b3b512301a48" instance_seqno: 1714572145819154) as {username='slave'} at 127.9.224.130:58551; Asking this server to re-register.
I20240501 14:02:35.681389 10766 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:35.681870 10904 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:35.681951 11550 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" instance_seqno: 1714572146262436) as {username='slave'} at 127.9.224.131:56503; Asking this server to re-register.
I20240501 14:02:35.682698 11038 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:35.682771 11550 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" instance_seqno: 1714572145337112) as {username='slave'} at 127.9.224.129:54957; Asking this server to re-register.
I20240501 14:02:35.683488 10766 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:35.683811 11551 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:35.684824 11550 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:35.685479 11551 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:35.815229 11591 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:35.815532 11591 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:35.817864 11591 leader_election.cc:288] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
W20240501 14:02:35.818902 11535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:35.826571 11535 leader_election.cc:334] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:35.830920 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 5 candidate_status { last_received { term: 4 index: 38 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6" is_pre_election: true
I20240501 14:02:35.831207 11230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47f54722b654655bfc871851b93abbc in term 4.
I20240501 14:02:35.831787 11538 leader_election.cc:302] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters: e048fbcc4e544adea37b6bd9e5ac8543
I20240501 14:02:35.832302 11591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 14:02:35.832489 11591 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:35.832578 11591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:02:35.836951 11591 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:35.837783 11591 leader_election.cc:288] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 5 election: Requested vote from peers e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
I20240501 14:02:35.838620 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 5 candidate_status { last_received { term: 4 index: 38 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6"
I20240501 14:02:35.838863 11230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 4 FOLLOWER]: Advancing to term 5
W20240501 14:02:35.839452 11535 leader_election.cc:334] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:35.841746 11230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47f54722b654655bfc871851b93abbc in term 5.
I20240501 14:02:35.842173 11538 leader_election.cc:302] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters: e048fbcc4e544adea37b6bd9e5ac8543
I20240501 14:02:35.842464 11591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:02:35.842909 11591 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 LEADER]: Becoming Leader. State: Replica: e47f54722b654655bfc871851b93abbc, State: Running, Role: LEADER
I20240501 14:02:35.843317 11591 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 4.38, Last appended by leader: 38, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:35.845943 11614 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47f54722b654655bfc871851b93abbc. Latest consensus state: current_term: 5 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:35.846246 11614 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:35.846853 11616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:35.849395 11616 catalog_manager.cc:669] Loaded metadata for table default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7]
I20240501 14:02:35.849850 11616 catalog_manager.cc:669] Loaded metadata for table default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd]
I20240501 14:02:35.850200 11616 catalog_manager.cc:669] Loaded metadata for table default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5]
I20240501 14:02:35.850539 11616 catalog_manager.cc:669] Loaded metadata for table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]
I20240501 14:02:35.850857 11616 catalog_manager.cc:669] Loaded metadata for table default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3]
I20240501 14:02:35.851169 11616 catalog_manager.cc:669] Loaded metadata for table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]
I20240501 14:02:35.851469 11616 catalog_manager.cc:669] Loaded metadata for table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]
I20240501 14:02:35.851801 11616 catalog_manager.cc:669] Loaded metadata for table default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323]
I20240501 14:02:35.852119 11616 catalog_manager.cc:669] Loaded metadata for table default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8]
I20240501 14:02:35.854141 11616 tablet_loader.cc:96] loaded metadata for tablet 2ddf192a1272427d899909c6bad20024 (table default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3])
I20240501 14:02:35.854451 11616 tablet_loader.cc:96] loaded metadata for tablet 37ae9258fe314b9b9271a8f25a085be2 (table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f])
I20240501 14:02:35.854755 11616 tablet_loader.cc:96] loaded metadata for tablet 543d48b213be415c9cd17bbf67e0bb5d (table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb])
I20240501 14:02:35.855017 11616 tablet_loader.cc:96] loaded metadata for tablet 552cf2dae53e4e81aad0bf5453eef53d (table default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5])
I20240501 14:02:35.855293 11616 tablet_loader.cc:96] loaded metadata for tablet 56ff9d28dcf24f24935f5c739316ff70 (table default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8])
I20240501 14:02:35.855543 11616 tablet_loader.cc:96] loaded metadata for tablet 645755771e714431ac3d429bfe6b01f0 (table default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd])
I20240501 14:02:35.855849 11616 tablet_loader.cc:96] loaded metadata for tablet 667ad182034b4dd2aaf55f1486d9318f (table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043])
I20240501 14:02:35.856417 11616 tablet_loader.cc:96] loaded metadata for tablet 81440735b20d43c2ae3d3913919a5bef (table default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7])
I20240501 14:02:35.856989 11616 tablet_loader.cc:96] loaded metadata for tablet dd9e4695e08d425aab0068cbcd673d8a (table default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323])
I20240501 14:02:35.857542 11616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:35.858568 11616 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:35.858651 11616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:35.859922 11616 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:35.860817 11616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Loaded TSK: 0
I20240501 14:02:35.861688 11616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:35.862560 11616 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 14:02:35.863548 11616 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240501 14:02:35.897354 11551 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36736:
table { table_name: "default.Table_d3a10d5d2bfb4854bd055ff3e0768f4e" } modify_external_catalogs: true
I20240501 14:02:35.897835 11550 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36736:
name: "default.Table_101396daa69f460997cbe7b34c1d12a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:35.898042 11547 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36736:
name: "default.Table_e314decf1e7044ed9f9796b96a898931"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:35.898253 11548 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36736:
table { table_name: "default.Table_30033735ecf54950a6eab1ae05ef1e55" } modify_external_catalogs: true
I20240501 14:02:35.898473 11548 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36736:
table { table_name: "default.Table_30033735ecf54950a6eab1ae05ef1e55" } modify_external_catalogs: true
I20240501 14:02:35.898763 11546 master_service.cc:946] ReplaceTablet: received request to replace tablet dfd4544926c04f7fa34419a8108e8d1e from {username='slave'} at 127.0.0.1:36736
I20240501 14:02:35.898823 11549 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36736:
name: "default.Table_a6090a9bb79448e7bd984f4fc6bb3217"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:35.899101 11543 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36736:
table { table_name: "default.Table_cc9ce1d1031f42789771ae5eb4a6df43" } new_table_name: "default.Table_b8695d7d9f484f03995f17c0bbc61ee3" modify_external_catalogs: true
I20240501 14:02:35.899425 11545 master_service.cc:946] ReplaceTablet: received request to replace tablet dfd4544926c04f7fa34419a8108e8d1e from {username='slave'} at 127.0.0.1:36736
I20240501 14:02:35.897728 11551 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36736:
table { table_name: "default.Table_d3a10d5d2bfb4854bd055ff3e0768f4e" } modify_external_catalogs: true
I20240501 14:02:35.900027 11546 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36736:
table { table_name: "default.Table_c9596eb80e944f919a8c85174a7080ca" } new_table_name: "default.Table_fa62350979764cedbfc470287d68ffe1" modify_external_catalogs: true
I20240501 14:02:35.900099 11545 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36736:
table { table_name: "default.Table_6d8f3ef2579a49df82d7ae9d6fbb3e2b" } new_table_name: "default.Table_6cc7bd2d05ec418ba2a930d28e0257b0" modify_external_catalogs: true
I20240501 14:02:35.900295 11542 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36736:
table { table_name: "default.Table_498fa9129a934033b9b363449b003bef" } new_table_name: "default.Table_ca677c5ce49e45d8ad3221682525d4c5" modify_external_catalogs: true
I20240501 14:02:35.900511 11544 master_service.cc:946] ReplaceTablet: received request to replace tablet 0496a42cfb744b25ad294d426ded937f from {username='slave'} at 127.0.0.1:36736
I20240501 14:02:35.901178 11544 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36736:
name: "default.Table_4801897b969f4fb8a4d46503057f4169"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20240501 14:02:35.905341 11604 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_498fa9129a934033b9b363449b003bef
*** Check failure stack trace: ***
@ 0x7f93181cedcd google::LogMessage::Fail() at ??:0
@ 0x7f93181d2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f93181ce7cc google::LogMessage::Flush() at ??:0
@ 0x7f93181cea11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f932e3e357e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f932e3e2261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f932e3e92bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f932e3e9139 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f931aebe01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f931e2436db start_thread at ??:0
@ 0x7f931708171f clone at ??:0
W20240501 14:02:36.091147 11045 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.224.190:34475)
W20240501 14:02:36.091574 11045 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.224.190:34475)
W20240501 14:02:36.091986 11045 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.224.190:34475)
W20240501 14:02:36.092774 11045 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.224.190:34475)
I20240501 14:02:37.944851 10114 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu pid 11152
I20240501 14:02:37.951963 11250 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:38.065491 11634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:38.065820 11634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:38.066051 10114 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu with pid 11152
I20240501 14:02:38.950038 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--webserver_interface=127.9.224.188
--webserver_port=35625
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 14:02:39.312471 11637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:39.312815 11637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:39.312937 11637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:39.320947 11637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:39.321138 11637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:39.321221 11637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:39.321305 11637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:39.331171 11637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.224.188
--webserver_port=35625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:39.332681 11637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:39.334800 11637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:39.346776 11642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:39.346755 11643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:39.347119 11645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:39.349517 11637 server_base.cc:1034] running on GCE node
I20240501 14:02:39.350410 11637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:39.352550 11637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:39.353838 11637 hybrid_clock.cc:648] HybridClock initialized: now 1714572159353742 us; error 59 us; skew 500 ppm
I20240501 14:02:39.354254 11637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:39.358011 11637 webserver.cc:460] Webserver started at http://127.9.224.188:35625/ using document root <none> and password file <none>
I20240501 14:02:39.358893 11637 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:39.359042 11637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:39.365314 11637 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:39.368548 11651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:39.370277 11637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:39.370502 11637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
uuid: "4163d103537745edb461e4dbdda724e6"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-gtmz"
I20240501 14:02:39.371515 11637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:39.392915 11637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:39.393910 11637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:39.394203 11637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:39.412431 11637 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.188:38257
I20240501 14:02:39.412449 11702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.188:38257 every 8 connection(s)
I20240501 14:02:39.414084 11637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
I20240501 14:02:39.420166 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 11637
I20240501 14:02:39.421269 11703 sys_catalog.cc:263] Verifying existing consensus state
I20240501 14:02:39.425176 11703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap starting.
I20240501 14:02:39.438294 11703 log.cc:824] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:39.467694 11703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=28 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:39.468245 11703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap complete.
I20240501 14:02:39.475093 11703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:39.475978 11703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4163d103537745edb461e4dbdda724e6, State: Initialized, Role: FOLLOWER
I20240501 14:02:39.476665 11703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 4.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:39.477649 11708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:39.477943 11708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:39.478703 11703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:39.491585 11722 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:39.491820 11722 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: loading cluster ID for follower catalog manager: success
I20240501 14:02:39.496484 11722 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: acquiring CA information for follower catalog manager: success
I20240501 14:02:39.499873 11722 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:39.501914 11637 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:02:39.509747 11637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:39.595743 10765 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:39.597333 10903 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:39.600400 11668 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" instance_seqno: 1714572145337112) as {username='slave'} at 127.9.224.129:60621; Asking this server to re-register.
I20240501 14:02:39.600350 11667 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "156c3e56589d4969b853b3b512301a48" instance_seqno: 1714572145819154) as {username='slave'} at 127.9.224.130:53227; Asking this server to re-register.
I20240501 14:02:39.601408 10903 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:39.602314 10765 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:39.604225 11667 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:39.605309 11668 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:39.628952 11037 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:39.630478 11667 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" instance_seqno: 1714572146262436) as {username='slave'} at 127.9.224.131:54929; Asking this server to re-register.
I20240501 14:02:39.631296 11037 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:39.633093 11667 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:39.877938 11708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:39.878265 11708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:39.880602 11708 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:39.881505 11652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:39.883080 11652 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:39.883317 11653 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:02:39.883448 11653 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:39.883919 11708 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 14:02:40.606998 11730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:40.607224 11730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:40.607954 11730 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:40.610149 11653 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:40.610441 11652 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:40.610630 11652 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:40.611009 11730 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 14:02:41.582988 11736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:41.583261 11736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:41.583930 11736 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:41.586325 11652 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:41.586566 11653 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:02:41.586714 11653 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:41.586961 11736 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 14:02:42.405838 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.190:34475
--webserver_interface=127.9.224.190
--webserver_port=35165
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 14:02:42.744223 11741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:42.744537 11741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:42.744665 11741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:42.752300 11741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:42.752418 11741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:42.752496 11741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:42.752564 11741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:42.762090 11741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.190:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.224.190
--webserver_port=35165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:42.763480 11741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:42.765381 11741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:42.776542 11746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:42.776934 11749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:42.777374 11747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:42.777766 11741 server_base.cc:1034] running on GCE node
I20240501 14:02:42.778558 11741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:42.780061 11741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:42.781280 11741 hybrid_clock.cc:648] HybridClock initialized: now 1714572162781210 us; error 45 us; skew 500 ppm
I20240501 14:02:42.781616 11741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:42.787860 11741 webserver.cc:460] Webserver started at http://127.9.224.190:35165/ using document root <none> and password file <none>
I20240501 14:02:42.788681 11741 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:42.788830 11741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:42.794775 11741 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 14:02:42.797927 11755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:42.799404 11741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:02:42.799610 11741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
uuid: "e47f54722b654655bfc871851b93abbc"
format_stamp: "Formatted at 2024-05-01 14:02:23 on dist-test-slave-gtmz"
I20240501 14:02:42.800544 11741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:42.823985 11741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:42.824880 11741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:42.825186 11741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:42.844231 11741 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.190:34475
I20240501 14:02:42.844251 11806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.190:34475 every 8 connection(s)
I20240501 14:02:42.845808 11741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-0/data/info.pb
I20240501 14:02:42.852046 11807 sys_catalog.cc:263] Verifying existing consensus state
I20240501 14:02:42.855289 11807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap starting.
I20240501 14:02:42.855947 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 11741
I20240501 14:02:42.868089 11807 log.cc:824] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:42.900467 11807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=38 ignored=0} inserts{seen=28 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:02:42.901144 11807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Bootstrap complete.
I20240501 14:02:42.907868 11807 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:42.908949 11807 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47f54722b654655bfc871851b93abbc, State: Initialized, Role: FOLLOWER
I20240501 14:02:42.909685 11807 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 5.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:42.910518 11811 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:42.910801 11811 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:42.911563 11807 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:42.931288 11825 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:42.931515 11825 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: loading cluster ID for follower catalog manager: success
I20240501 14:02:42.935286 11825 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: acquiring CA information for follower catalog manager: success
I20240501 14:02:42.938696 11825 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:42.935089 11741 master_runner.cc:186] Error getting master registration for 127.9.224.189:42871: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111),
I20240501 14:02:42.942519 11741 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:43.216455 10766 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:43.216413 11038 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:43.220430 11771 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" instance_seqno: 1714572145337112) as {username='slave'} at 127.9.224.129:37705; Asking this server to re-register.
I20240501 14:02:43.220410 11772 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" instance_seqno: 1714572146262436) as {username='slave'} at 127.9.224.131:53491; Asking this server to re-register.
I20240501 14:02:43.221436 10766 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:43.221535 11038 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:43.223086 10904 heartbeater.cc:347] Connected to a master server at 127.9.224.190:34475
I20240501 14:02:43.223717 11772 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:43.223851 11770 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:43.225389 11770 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "156c3e56589d4969b853b3b512301a48" instance_seqno: 1714572145819154) as {username='slave'} at 127.9.224.130:48661; Asking this server to re-register.
I20240501 14:02:43.226037 10904 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:43.227710 11770 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:43.237695 11831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:43.237913 11831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:43.238602 11831 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:43.243149 11652 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:43.248327 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4163d103537745edb461e4dbdda724e6" candidate_term: 6 candidate_status { last_received { term: 4 index: 38 } } ignore_live_leader: false dest_uuid: "e47f54722b654655bfc871851b93abbc" is_pre_election: true
I20240501 14:02:43.249215 11782 raft_consensus.cc:2396] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4163d103537745edb461e4dbdda724e6 for term 6 because replica has last-logged OpId of term: 5 index: 39, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 38.
I20240501 14:02:43.250203 11653 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:43.250561 11831 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 14:02:43.463773 11836 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:43.464076 11836 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:43.465973 11836 leader_election.cc:288] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
W20240501 14:02:43.467140 11756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:43.468770 11756 leader_election.cc:334] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:43.473400 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 6 candidate_status { last_received { term: 5 index: 39 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6" is_pre_election: true
I20240501 14:02:43.473843 11678 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47f54722b654655bfc871851b93abbc in term 5.
I20240501 14:02:43.474449 11759 leader_election.cc:302] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters: e048fbcc4e544adea37b6bd9e5ac8543
I20240501 14:02:43.474927 11836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 14:02:43.475077 11836 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:43.475178 11836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:02:43.479414 11836 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:43.480195 11836 leader_election.cc:288] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 6 election: Requested vote from peers e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871), 4163d103537745edb461e4dbdda724e6 (127.9.224.188:38257)
I20240501 14:02:43.480931 11678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47f54722b654655bfc871851b93abbc" candidate_term: 6 candidate_status { last_received { term: 5 index: 39 } } ignore_live_leader: false dest_uuid: "4163d103537745edb461e4dbdda724e6"
I20240501 14:02:43.481197 11678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 5 FOLLOWER]: Advancing to term 6
W20240501 14:02:43.481781 11756 leader_election.cc:334] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:43.485781 11678 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47f54722b654655bfc871851b93abbc in term 6.
I20240501 14:02:43.486382 11759 leader_election.cc:302] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4163d103537745edb461e4dbdda724e6, e47f54722b654655bfc871851b93abbc; no voters: e048fbcc4e544adea37b6bd9e5ac8543
I20240501 14:02:43.486726 11836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:02:43.487349 11836 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [term 6 LEADER]: Becoming Leader. State: Replica: e47f54722b654655bfc871851b93abbc, State: Running, Role: LEADER
I20240501 14:02:43.487906 11836 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 5.39, Last appended by leader: 39, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:43.490777 11841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47f54722b654655bfc871851b93abbc. Latest consensus state: current_term: 6 leader_uuid: "e47f54722b654655bfc871851b93abbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:43.491125 11841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:43.491806 11843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:43.494483 11843 catalog_manager.cc:669] Loaded metadata for table default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7]
I20240501 14:02:43.494956 11843 catalog_manager.cc:669] Loaded metadata for table default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd]
I20240501 14:02:43.495311 11843 catalog_manager.cc:669] Loaded metadata for table default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5]
I20240501 14:02:43.495640 11843 catalog_manager.cc:669] Loaded metadata for table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb]
I20240501 14:02:43.495975 11843 catalog_manager.cc:669] Loaded metadata for table default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3]
I20240501 14:02:43.496294 11843 catalog_manager.cc:669] Loaded metadata for table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043]
I20240501 14:02:43.496603 11843 catalog_manager.cc:669] Loaded metadata for table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f]
I20240501 14:02:43.496937 11843 catalog_manager.cc:669] Loaded metadata for table default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323]
I20240501 14:02:43.497295 11843 catalog_manager.cc:669] Loaded metadata for table default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8]
I20240501 14:02:43.499322 11843 tablet_loader.cc:96] loaded metadata for tablet 2ddf192a1272427d899909c6bad20024 (table default.table_6d8f3ef2579a49df82d7ae9d6fbb3e2b [id=7115aaf817d14cd9ab186e9e9d789ca3])
I20240501 14:02:43.499653 11843 tablet_loader.cc:96] loaded metadata for tablet 37ae9258fe314b9b9271a8f25a085be2 (table default.table_c9596eb80e944f919a8c85174a7080ca [id=8e7509fb5fb74cc1a065864a5721040f])
I20240501 14:02:43.499954 11843 tablet_loader.cc:96] loaded metadata for tablet 543d48b213be415c9cd17bbf67e0bb5d (table default.table_e4a61f4822d448919e6decf48ba63a8d [id=3eeae6bd61554f2f8624397bf2db91cb])
I20240501 14:02:43.500223 11843 tablet_loader.cc:96] loaded metadata for tablet 552cf2dae53e4e81aad0bf5453eef53d (table default.table_d67cd2442dc24ffc8d4dc86e19a50315 [id=3a6f7d0bf2794a7e8717c2355971c0a5])
I20240501 14:02:43.500506 11843 tablet_loader.cc:96] loaded metadata for tablet 56ff9d28dcf24f24935f5c739316ff70 (table default.table_740f0f0aef9c46919f5658f6d2be21e3 [id=ee51129ad60e40bcb0a839c84a33a5e8])
I20240501 14:02:43.500731 11843 tablet_loader.cc:96] loaded metadata for tablet 645755771e714431ac3d429bfe6b01f0 (table default.table_30033735ecf54950a6eab1ae05ef1e55 [id=35433d009fbb4c27b7fa21916b08ddfd])
I20240501 14:02:43.500967 11843 tablet_loader.cc:96] loaded metadata for tablet 667ad182034b4dd2aaf55f1486d9318f (table default.table_498fa9129a934033b9b363449b003bef [id=7c5ba258d7764086a1e0168310f43043])
I20240501 14:02:43.501542 11843 tablet_loader.cc:96] loaded metadata for tablet 81440735b20d43c2ae3d3913919a5bef (table default.table_cc9ce1d1031f42789771ae5eb4a6df43 [id=0ecb2a95ea54458d87a8e0361bd772b7])
I20240501 14:02:43.502171 11843 tablet_loader.cc:96] loaded metadata for tablet dd9e4695e08d425aab0068cbcd673d8a (table default.table_d3a10d5d2bfb4854bd055ff3e0768f4e [id=c425cd655f0e4e26b6a30c819c63f323])
I20240501 14:02:43.502779 11843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:43.503829 11843 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:43.503907 11843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:43.505322 11843 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:43.506323 11843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e47f54722b654655bfc871851b93abbc: Loaded TSK: 0
I20240501 14:02:43.507205 11843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:43.508186 11843 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 14:02:43.509114 11843 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240501 14:02:43.612493 11772 master_service.cc:946] ReplaceTablet: received request to replace tablet fd6038c451a846c6b6541c0ac115ca12 from {username='slave'} at 127.0.0.1:40516
I20240501 14:02:43.613073 11770 master_service.cc:946] ReplaceTablet: received request to replace tablet b747db1c8b2b4921b3a1d23012f8b3a8 from {username='slave'} at 127.0.0.1:40516
I20240501 14:02:43.613240 11771 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40516:
name: "default.Table_557e6f54359d445eab8bcfb616e974ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:43.614080 11763 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40516:
table { table_name: "default.Table_30033735ecf54950a6eab1ae05ef1e55" } modify_external_catalogs: true
I20240501 14:02:43.613936 11770 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40516:
name: "default.Table_a6090a9bb79448e7bd984f4fc6bb3217"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:43.614243 11764 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40516:
name: "default.Table_e314decf1e7044ed9f9796b96a898931"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:43.614459 11763 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40516:
table { table_name: "default.Table_30033735ecf54950a6eab1ae05ef1e55" } modify_external_catalogs: true
I20240501 14:02:43.614966 11765 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40516:
table { table_name: "default.Table_d67cd2442dc24ffc8d4dc86e19a50315" } modify_external_catalogs: true
I20240501 14:02:43.615252 11765 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40516:
table { table_name: "default.Table_d67cd2442dc24ffc8d4dc86e19a50315" } modify_external_catalogs: true
I20240501 14:02:43.615213 11769 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40516:
name: "default.Table_f616bd9bd35f4659896ecc727f466014"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:43.615906 11766 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40516:
name: "default.Table_a67258cb839442b69e96ce75fc4c281b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:43.616103 11768 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40516:
name: "default.Table_1a9db37a35094b309731f45b45ef8113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:02:43.616463 11767 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40516:
table { table_name: "default.Table_740f0f0aef9c46919f5658f6d2be21e3" } modify_external_catalogs: true
I20240501 14:02:43.616736 11767 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40516:
table { table_name: "default.Table_740f0f0aef9c46919f5658f6d2be21e3" } modify_external_catalogs: true
I20240501 14:02:43.616851 11772 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40516:
table { table_name: "default.Table_d3a10d5d2bfb4854bd055ff3e0768f4e" } modify_external_catalogs: true
I20240501 14:02:43.617061 11772 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40516:
table { table_name: "default.Table_d3a10d5d2bfb4854bd055ff3e0768f4e" } modify_external_catalogs: true
F20240501 14:02:43.620826 11824 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_498fa9129a934033b9b363449b003bef
*** Check failure stack trace: ***
@ 0x7fca16fb2dcd google::LogMessage::Fail() at ??:0
@ 0x7fca16fb6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fca16fb27cc google::LogMessage::Flush() at ??:0
@ 0x7fca16fb2a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fca2d1c757e kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fca2d1c6261 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fca2d1cd2bc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fca2d1cd139 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fca19ca201e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fca1d0276db start_thread at ??:0
@ 0x7fca15e6571f clone at ??:0
W20240501 14:02:43.821599 10792 connection.cc:535] client connection to 127.9.224.190:34475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:43.821873 10926 connection.cc:535] client connection to 127.9.224.190:34475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:43.821987 11045 connection.cc:535] client connection to 127.9.224.190:34475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:43.822419 10654 connection.cc:535] client connection to 127.9.224.190:34475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:43.823071 11045 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.9.224.190:34475)
W20240501 14:02:43.823684 11045 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.224.190:34475)
W20240501 14:02:43.823974 11045 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.9.224.190:34475)
W20240501 14:02:43.824360 11045 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.9.224.190:34475)
I20240501 14:02:44.511432 11637 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:02:44.511658 11637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:45.074340 10114 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu pid 11637
I20240501 14:02:45.081192 11698 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:45.188798 10114 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu with pid 11637
I20240501 14:02:45.990796 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--webserver_interface=127.9.224.188
--webserver_port=35625
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 14:02:46.331707 11867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:46.332031 11867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:46.332120 11867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:46.339793 11867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:46.339926 11867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:46.339972 11867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:46.340035 11867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:46.351533 11867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.224.188
--webserver_port=35625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:46.353206 11867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:46.355453 11867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:46.367611 11872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:46.368009 11873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:46.368366 11875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:46.368501 11867 server_base.cc:1034] running on GCE node
I20240501 14:02:46.369416 11867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:46.370868 11867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:46.372088 11867 hybrid_clock.cc:648] HybridClock initialized: now 1714572166372021 us; error 49 us; skew 500 ppm
I20240501 14:02:46.372412 11867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:46.374840 11867 webserver.cc:460] Webserver started at http://127.9.224.188:35625/ using document root <none> and password file <none>
I20240501 14:02:46.375531 11867 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:46.375649 11867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:46.381412 11867 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:46.384508 11881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:46.386073 11867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:46.386250 11867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
uuid: "4163d103537745edb461e4dbdda724e6"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-gtmz"
I20240501 14:02:46.387143 11867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:46.415661 11867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:46.416536 11867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:46.416802 11867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:46.434437 11867 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.188:38257
I20240501 14:02:46.434458 11932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.188:38257 every 8 connection(s)
I20240501 14:02:46.435971 11867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
I20240501 14:02:46.440546 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 11867
I20240501 14:02:46.443011 11933 sys_catalog.cc:263] Verifying existing consensus state
I20240501 14:02:46.446849 11933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap starting.
I20240501 14:02:46.459334 11933 log.cc:824] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:46.490867 11933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=28 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:46.491449 11933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap complete.
I20240501 14:02:46.498407 11933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:46.499344 11933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4163d103537745edb461e4dbdda724e6, State: Initialized, Role: FOLLOWER
I20240501 14:02:46.499961 11933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 4.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:46.500737 11937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:46.501109 11937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:46.501824 11933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:46.516927 11951 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:46.517063 11951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: loading cluster ID for follower catalog manager: success
I20240501 14:02:46.511848 11867 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:02:46.519428 11867 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:46.521739 11951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: acquiring CA information for follower catalog manager: success
I20240501 14:02:46.525002 11951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:46.700340 11037 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:46.702435 10765 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:46.705160 11897 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" instance_seqno: 1714572145337112) as {username='slave'} at 127.9.224.129:48059; Asking this server to re-register.
I20240501 14:02:46.705111 11898 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" instance_seqno: 1714572146262436) as {username='slave'} at 127.9.224.131:54029; Asking this server to re-register.
I20240501 14:02:46.705935 11037 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:46.706135 10765 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:46.708221 11897 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:46.708442 11898 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:46.715260 10903 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:46.716782 11898 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "156c3e56589d4969b853b3b512301a48" instance_seqno: 1714572145819154) as {username='slave'} at 127.9.224.130:57633; Asking this server to re-register.
I20240501 14:02:46.717459 10903 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:46.719192 11898 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:46.925323 11937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:46.925556 11937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:46.927313 11937 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:46.927964 11883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:46.929471 11882 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:46.929649 11883 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:02:46.929811 11883 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:46.930230 11937 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:47.537990 11959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:47.538224 11959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:47.538981 11959 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:47.540830 11883 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:47.541013 11882 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:47.541199 11882 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:47.541464 11959 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:48.309831 11965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:48.310029 11965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:48.310693 11965 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:48.312820 11882 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:48.313023 11883 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:02:48.313174 11883 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:48.313449 11965 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:48.811012 10114 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu pid 11867
I20240501 14:02:48.825103 11928 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:48.934185 10114 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu with pid 11867
I20240501 14:02:49.747530 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--webserver_interface=127.9.224.188
--webserver_port=35625
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 14:02:50.093076 11975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:50.093444 11975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:50.093573 11975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:50.101634 11975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:50.101771 11975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:50.101845 11975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:50.101922 11975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:50.111667 11975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.224.188
--webserver_port=35625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:50.113088 11975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:50.114871 11975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:50.125999 11980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:50.126305 11981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:50.126926 11983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:50.127599 11975 server_base.cc:1034] running on GCE node
I20240501 14:02:50.128415 11975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:50.129953 11975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:50.131211 11975 hybrid_clock.cc:648] HybridClock initialized: now 1714572170131133 us; error 53 us; skew 500 ppm
I20240501 14:02:50.131565 11975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:50.134063 11975 webserver.cc:460] Webserver started at http://127.9.224.188:35625/ using document root <none> and password file <none>
I20240501 14:02:50.134837 11975 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:50.134974 11975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:50.140914 11975 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 14:02:50.144076 11989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:50.145726 11975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:50.145929 11975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
uuid: "4163d103537745edb461e4dbdda724e6"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-gtmz"
I20240501 14:02:50.146855 11975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:50.177510 11975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:50.178424 11975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:50.178711 11975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:50.197556 11975 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.188:38257
I20240501 14:02:50.197569 12040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.188:38257 every 8 connection(s)
I20240501 14:02:50.199370 11975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
I20240501 14:02:50.208151 12041 sys_catalog.cc:263] Verifying existing consensus state
I20240501 14:02:50.208674 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 11975
I20240501 14:02:50.214517 12041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap starting.
I20240501 14:02:50.228818 10765 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:50.229837 12041 log.cc:824] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:50.230626 11037 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:50.238718 10903 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:50.263267 12041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=28 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:50.263829 12041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap complete.
I20240501 14:02:50.270535 12041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:50.271394 12041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4163d103537745edb461e4dbdda724e6, State: Initialized, Role: FOLLOWER
I20240501 14:02:50.272085 12041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 4.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:50.272956 12046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:50.273370 12046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:50.274293 12041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:50.288175 12060 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:50.288365 12060 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: loading cluster ID for follower catalog manager: success
I20240501 14:02:50.295719 12060 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: acquiring CA information for follower catalog manager: success
I20240501 14:02:50.299458 12060 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:50.293076 11975 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:02:50.301486 11975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:50.734293 12005 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" instance_seqno: 1714572146262436) as {username='slave'} at 127.9.224.131:46285; Asking this server to re-register.
I20240501 14:02:50.734239 12006 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" instance_seqno: 1714572145337112) as {username='slave'} at 127.9.224.129:49091; Asking this server to re-register.
I20240501 14:02:50.734949 12046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:50.735118 11037 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:50.735239 10765 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:50.735252 12046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:50.737134 12006 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:50.737290 12005 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:50.737686 12046 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:50.738523 11991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:50.739957 11991 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:50.740190 11990 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:50.740330 11990 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:50.740752 12046 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:50.741921 12005 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "156c3e56589d4969b853b3b512301a48" instance_seqno: 1714572145819154) as {username='slave'} at 127.9.224.130:48853; Asking this server to re-register.
I20240501 14:02:50.742498 10903 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:50.743912 12005 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:51.273639 12067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:51.273913 12067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:51.274711 12067 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:51.276844 11991 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:51.277405 11990 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:51.277547 11990 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:51.277953 12067 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:52.388280 12072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:52.388496 12072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:52.389221 12072 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:52.391639 11991 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:52.391855 11990 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:52.392000 11990 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:52.392266 12072 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:52.642709 10114 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu pid 11975
I20240501 14:02:52.650355 12036 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:52.754252 10114 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu with pid 11975
I20240501 14:02:53.681771 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--webserver_interface=127.9.224.188
--webserver_port=35625
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 14:02:54.029860 12081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:54.030193 12081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:54.030310 12081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:54.038875 12081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:54.039005 12081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:54.039048 12081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:54.039105 12081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:54.049966 12081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.224.188
--webserver_port=35625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:54.051380 12081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:54.053300 12081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:54.064935 12088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:54.065279 12091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:54.065615 12089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:54.071033 12081 server_base.cc:1034] running on GCE node
I20240501 14:02:54.072089 12081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:54.073935 12081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:54.075238 12081 hybrid_clock.cc:648] HybridClock initialized: now 1714572174075170 us; error 43 us; skew 500 ppm
I20240501 14:02:54.075699 12081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:54.085168 12081 webserver.cc:460] Webserver started at http://127.9.224.188:35625/ using document root <none> and password file <none>
I20240501 14:02:54.086019 12081 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:54.086169 12081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:54.092269 12081 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 14:02:54.095970 12097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:54.097692 12081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:02:54.097935 12081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
uuid: "4163d103537745edb461e4dbdda724e6"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-gtmz"
I20240501 14:02:54.098855 12081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:54.116863 12081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:54.118084 12081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:54.118438 12081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:54.138082 12081 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.188:38257
I20240501 14:02:54.138093 12148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.188:38257 every 8 connection(s)
I20240501 14:02:54.139789 12081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
I20240501 14:02:54.141767 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 12081
I20240501 14:02:54.147873 12149 sys_catalog.cc:263] Verifying existing consensus state
I20240501 14:02:54.153672 12149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap starting.
I20240501 14:02:54.170446 12149 log.cc:824] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:54.204805 12149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=28 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:54.205442 12149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap complete.
I20240501 14:02:54.212397 12149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:54.213394 12149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4163d103537745edb461e4dbdda724e6, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.214087 12149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 4.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:54.215315 12154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:54.215870 12154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:54.216511 12149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:54.237244 12168 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:54.237457 12168 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: loading cluster ID for follower catalog manager: success
I20240501 14:02:54.231393 12081 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:02:54.238994 12081 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:54.242125 12168 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: acquiring CA information for follower catalog manager: success
I20240501 14:02:54.245828 12168 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:54.266564 10765 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:54.266619 11037 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:54.269989 12113 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" instance_seqno: 1714572146262436) as {username='slave'} at 127.9.224.131:42351; Asking this server to re-register.
I20240501 14:02:54.269961 12114 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" instance_seqno: 1714572145337112) as {username='slave'} at 127.9.224.129:50277; Asking this server to re-register.
I20240501 14:02:54.270924 10765 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:54.271031 11037 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:54.273074 12114 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:54.273236 12113 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:54.281769 10903 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:54.283428 12113 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "156c3e56589d4969b853b3b512301a48" instance_seqno: 1714572145819154) as {username='slave'} at 127.9.224.130:37741; Asking this server to re-register.
I20240501 14:02:54.284154 10903 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:54.286010 12113 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:54.591370 12154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:54.591722 12154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:54.593853 12154 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:54.594727 12099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:54.596390 12098 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:54.597589 12099 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:02:54.598453 12099 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:54.599236 12154 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:55.146792 12175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:55.147040 12175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:55.147866 12175 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:55.150473 12099 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:55.150677 12098 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:55.150848 12098 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:55.151108 12175 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:55.984506 12181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:55.984753 12181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:55.985482 12181 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:55.987684 12099 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:55.987866 12098 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:55.988013 12098 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:55.988306 12181 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:56.364825 10114 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu pid 12081
I20240501 14:02:56.372081 12144 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:56.477865 10114 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu with pid 12081
I20240501 14:02:57.334554 10114 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
/tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--webserver_interface=127.9.224.188
--webserver_port=35625
--builtin_ntp_servers=127.9.224.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 14:02:57.669013 12191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:57.669348 12191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:57.669462 12191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:57.677119 12191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:57.677234 12191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:57.677299 12191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:57.677367 12191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:57.686810 12191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.224.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37661
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.224.188:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.224.188
--webserver_port=35625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:57.688210 12191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:57.690011 12191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:57.702376 12196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:57.702708 12199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:57.703208 12197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:57.703711 12191 server_base.cc:1034] running on GCE node
I20240501 14:02:57.704635 12191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:57.706328 12191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:57.707556 12191 hybrid_clock.cc:648] HybridClock initialized: now 1714572177707487 us; error 48 us; skew 500 ppm
I20240501 14:02:57.707948 12191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:57.710304 12191 webserver.cc:460] Webserver started at http://127.9.224.188:35625/ using document root <none> and password file <none>
I20240501 14:02:57.711016 12191 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:57.711130 12191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:57.716441 12191 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:57.719203 12205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:57.720551 12191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:57.720721 12191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data,/tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
uuid: "4163d103537745edb461e4dbdda724e6"
format_stamp: "Formatted at 2024-05-01 14:02:24 on dist-test-slave-gtmz"
I20240501 14:02:57.721611 12191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:57.739344 12191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:57.740239 12191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:57.740532 12191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:57.758536 12191 rpc_server.cc:307] RPC server started. Bound to: 127.9.224.188:38257
I20240501 14:02:57.758556 12257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.224.188:38257 every 8 connection(s)
I20240501 14:02:57.759985 12191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmQb6Mr/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714572127973417-10114-0/minicluster-data/master-2/data/info.pb
I20240501 14:02:57.763667 10114 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmQb6Mr/build/asan/bin/kudu as pid 12191
I20240501 14:02:57.767657 12258 sys_catalog.cc:263] Verifying existing consensus state
I20240501 14:02:57.773042 12258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap starting.
I20240501 14:02:57.785694 12258 log.cc:824] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:57.795442 11037 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:57.795514 10765 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:57.804394 10903 heartbeater.cc:347] Connected to a master server at 127.9.224.188:38257
I20240501 14:02:57.818740 12258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=28 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:57.819227 12258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: Bootstrap complete.
I20240501 14:02:57.825572 12258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:57.826426 12258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4163d103537745edb461e4dbdda724e6, State: Initialized, Role: FOLLOWER
I20240501 14:02:57.827087 12258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 4.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:57.827914 12263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } } }
I20240501 14:02:57.828194 12263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 14:02:57.828832 12258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:57.868729 12277 catalog_manager.cc:1259] Loaded cluster ID: 598019b5e51d4567bf5a30b1ca667595
I20240501 14:02:57.868858 12277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: loading cluster ID for follower catalog manager: success
I20240501 14:02:57.873518 12277 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: acquiring CA information for follower catalog manager: success
I20240501 14:02:57.876772 12277 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 14:02:57.874225 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:02:57.879284 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:02:58.248706 12263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:58.249058 12263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:58.251133 12263 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:58.251801 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:58.253343 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:58.253551 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:58.253763 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:58.254242 12263 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:58.300216 12222 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e09841056a14c29be2546cc1d0b4643" instance_seqno: 1714572145337112) as {username='slave'} at 127.9.224.129:51129; Asking this server to re-register.
I20240501 14:02:58.300171 12223 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c1d9952ad6a24cecbcde3d02a59c9a75" instance_seqno: 1714572146262436) as {username='slave'} at 127.9.224.131:39571; Asking this server to re-register.
I20240501 14:02:58.300886 11037 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:58.300967 10765 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:58.303010 12222 ts_manager.cc:194] Registered new tserver with Master: c1d9952ad6a24cecbcde3d02a59c9a75 (127.9.224.131:35371)
I20240501 14:02:58.303130 12223 ts_manager.cc:194] Registered new tserver with Master: 5e09841056a14c29be2546cc1d0b4643 (127.9.224.129:38539)
I20240501 14:02:58.307471 12222 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "156c3e56589d4969b853b3b512301a48" instance_seqno: 1714572145819154) as {username='slave'} at 127.9.224.130:41235; Asking this server to re-register.
I20240501 14:02:58.308019 10903 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:58.309408 12222 ts_manager.cc:194] Registered new tserver with Master: 156c3e56589d4969b853b3b512301a48 (127.9.224.130:38403)
I20240501 14:02:58.857820 12283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:58.858042 12283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:58.858716 12283 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:58.861124 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:02:58.861301 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:02:58.861542 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:58.861846 12283 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:02:59.598224 12288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:59.598444 12288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:02:59.599151 12288 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:02:59.601513 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:02:59.601699 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:02:59.601832 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:02:59.602113 12288 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:00.132655 10114 master-stress-test.cc:578] Killing test thread 1/25
I20240501 14:03:01.177412 12294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:01.177634 12294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:01.178294 12294 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:01.180862 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:01.181095 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:03:01.181305 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:01.181649 12294 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:02.598208 12301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:02.598448 12301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:02.599102 12301 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:02.601593 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:03:02.601878 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:03:02.602142 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:02.602507 12301 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:02.880672 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:02.880862 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:05.746532 12306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:05.746764 12306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:05.747479 12306 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:05.749075 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:03:05.753361 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:05.753625 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:03:05.753777 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:05.754102 12306 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:07.882665 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:07.882836 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:10.502441 12319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:10.502676 12319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:10.503407 12319 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:10.505848 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:10.506121 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:03:10.506258 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:10.506695 12319 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:12.884477 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:12.884718 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:16.925351 12334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:16.925617 12334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:16.926553 12334 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:16.927529 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:03:16.928660 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:16.928865 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:03:16.928997 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:16.929235 12334 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:17.886279 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:17.886444 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:22.888031 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:22.888227 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:25.961968 12354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:25.962258 12354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:25.963006 12354 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:25.964241 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:03:25.965515 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:25.965727 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:03:25.965867 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:25.966279 12354 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240501 14:03:26.695811 10767 heartbeater.cc:644] Failed to heartbeat to 127.9.224.189:42871 (133 consecutive failures): Network error: Failed to ping master at 127.9.224.189:42871: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:03:26.938562 10904 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (86 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:26.938645 11038 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (86 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:03:27.894707 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:27.894874 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:27.975461 12360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:27.975685 12360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:27.976351 12360 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:27.977983 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:27.978179 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:03:27.978309 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:27.978571 12360 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:32.896381 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:32.896548 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:37.898227 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:37.898427 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:39.801395 12378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:39.801604 12378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:39.802366 12378 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:39.803555 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:03:39.804878 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:03:39.805092 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:03:39.805214 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:39.805483 12378 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:41.036695 12384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:41.036933 12384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:41.037686 12384 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:41.039606 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:41.039796 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:03:41.039925 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:41.040202 12384 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:42.901815 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:42.902012 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:47.903621 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:47.903815 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:48.289381 12400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:48.289626 12400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:48.290398 12400 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:48.291038 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:03:48.292285 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:48.292460 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:03:48.292594 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:48.292863 12400 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:52.905819 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:52.906005 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:03:55.447422 12417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:55.447667 12417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:03:55.448367 12417 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:03:55.450007 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:03:55.451534 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:03:55.451915 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:03:55.452136 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:03:55.452662 12417 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:03:57.907673 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:03:57.907923 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:02.909564 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:02.909742 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:07.675976 12442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:07.676187 12442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:04:07.677112 12442 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:04:07.679665 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:04:07.680933 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:04:07.681277 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:04:07.681535 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:04:07.681850 12442 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:04:07.911285 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:07.911475 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:12.913687 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:12.914018 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:14.559455 12454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:14.559669 12454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:04:14.560451 12454 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:04:14.561821 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:04:14.563313 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:04:14.563508 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:04:14.563637 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:04:14.563972 12454 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:04:17.915803 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:17.915998 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:22.917713 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:22.917899 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 14:04:26.859067 10767 heartbeater.cc:644] Failed to heartbeat to 127.9.224.189:42871 (253 consecutive failures): Network error: Failed to ping master at 127.9.224.189:42871: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:04:27.102727 10904 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (206 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:04:27.103022 11038 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (206 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:04:27.919366 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:27.919559 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:32.715617 12483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:32.715842 12483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:04:32.716748 12483 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:04:32.717783 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:04:32.720140 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:04:32.720307 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:04:32.720441 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:04:32.720914 12483 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:04:32.921275 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:32.921519 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:37.923447 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:37.923651 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:42.925369 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:42.925554 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:44.249027 12503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:44.249274 12503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:04:44.250082 12503 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:04:44.251889 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:04:44.253502 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:04:44.253680 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:04:44.253808 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:04:44.254048 12503 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:04:47.927636 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:47.927841 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:52.929843 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:52.930107 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:57.931638 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:04:57.931824 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:04:57.943378 12212 maintenance_manager.cc:419] P 4163d103537745edb461e4dbdda724e6: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033365
I20240501 14:04:57.988759 12211 maintenance_manager.cc:643] P 4163d103537745edb461e4dbdda724e6: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.025s sys 0.000s Metrics: {"bytes_written":12692,"cfile_init":1,"dirs.queue_time_us":411,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1108,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":591,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":28,"thread_start_us":494,"threads_started":2,"wal-append.queue_time_us":326}
I20240501 14:04:57.990027 12212 maintenance_manager.cc:419] P 4163d103537745edb461e4dbdda724e6: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1451 bytes on disk
I20240501 14:04:57.990768 12211 maintenance_manager.cc:643] P 4163d103537745edb461e4dbdda724e6: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240501 14:05:02.933358 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:02.933524 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:03.645586 12584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:03.645784 12584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:05:03.646689 12584 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:05:03.647708 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:05:03.648857 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:05:03.649098 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:05:03.649202 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:05:03.649482 12584 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:05:07.935318 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:07.935518 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:12.937093 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:12.937307 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:17.938772 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:17.938956 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:22.940604 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:22.940778 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:23.117827 12684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:23.118073 12684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:05:23.118968 12684 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:05:23.119594 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:05:23.120777 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:05:23.120936 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:05:23.121104 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:05:23.121340 12684 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240501 14:05:27.005409 10767 heartbeater.cc:644] Failed to heartbeat to 127.9.224.189:42871 (373 consecutive failures): Network error: Failed to ping master at 127.9.224.189:42871: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:05:27.254640 11038 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (326 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:05:27.254859 10904 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (326 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:05:27.942205 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:27.942389 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:32.944087 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:32.944264 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:36.927177 12755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:36.927457 12755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:05:36.928418 12755 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:05:36.929611 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:05:36.941102 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:05:36.942756 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:05:36.942937 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:05:36.943248 12755 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:05:37.946321 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:37.946523 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:42.948652 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:42.948879 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:43.415724 12790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:43.416031 12790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:05:43.416776 12790 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:05:43.417366 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:05:43.419006 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:05:43.419343 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:05:43.419518 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:05:43.419834 12790 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:05:47.950848 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:47.951155 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:49.485333 12822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:49.485576 12822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:05:49.486346 12822 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:05:49.489337 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:05:49.491205 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:05:49.491540 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:05:49.491653 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:05:49.492338 12822 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:05:52.953236 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:52.953454 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:57.955555 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:05:57.955770 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:05:58.243346 12869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:58.243578 12869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:05:58.244323 12869 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:05:58.244925 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:05:58.246402 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:05:58.246590 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:05:58.246717 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:05:58.247001 12869 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:06:01.012930 12884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:01.013267 12884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:06:01.014390 12884 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:06:01.017704 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:06:01.017894 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:06:01.018041 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:06:01.018797 12884 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:06:02.957306 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:02.957463 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:03.560262 12899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:03.560456 12899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:06:03.561194 12899 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:06:03.561998 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:06:03.563367 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:06:03.563542 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:06:03.563655 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:06:03.563936 12899 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:06:07.959045 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:07.959216 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:12.961046 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:12.961270 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:15.222927 12958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:15.223143 12958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:06:15.223820 12958 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:06:15.224690 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:06:15.225823 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:06:15.225984 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:06:15.226105 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:06:15.226313 12958 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:06:17.962687 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:17.962859 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:20.221694 12984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:20.221892 12984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:06:20.222538 12984 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:06:20.224365 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:06:20.224539 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:06:20.224658 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:06:20.224900 12984 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:06:22.964264 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:22.964448 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:24.096212 13006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:24.096401 13006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:06:24.097244 13006 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:06:24.098146 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:06:24.099500 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:06:24.099648 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:06:24.099772 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:06:24.100009 13006 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240501 14:06:27.160892 10767 heartbeater.cc:644] Failed to heartbeat to 127.9.224.189:42871 (493 consecutive failures): Network error: Failed to ping master at 127.9.224.189:42871: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:06:27.412736 10904 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (446 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:06:27.413301 11038 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (446 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:06:27.965931 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:27.966117 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:28.360699 13028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:28.360930 13028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:06:28.361579 13028 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:06:28.363162 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:06:28.363373 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:06:28.363488 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:06:28.363737 13028 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:06:32.967577 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:32.967753 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:37.969312 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:37.969472 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:42.970921 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:42.971104 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:47.906755 13114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:47.907012 13114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:06:47.907933 13114 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:06:47.908917 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:06:47.910125 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:06:47.910279 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:06:47.910391 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:06:47.910640 13114 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:06:47.972111 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:47.972287 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:52.973980 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:52.974138 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:57.975543 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:06:57.975710 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:06:58.263186 13162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:58.263381 13162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:06:58.264081 13162 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:06:58.264642 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:06:58.265908 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:06:58.266067 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:06:58.266175 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:06:58.266381 13162 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:07:00.212105 13172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:00.212322 13172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:07:00.213023 13172 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:07:00.214938 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:00.215129 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:07:00.215240 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:07:00.215490 13172 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:07:02.977144 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:07:02.977322 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:07:07.978906 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:07:07.979079 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:07:11.165233 13223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:11.165455 13223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:07:11.166150 13223 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:07:11.167106 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:07:11.168251 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:11.168467 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:07:11.168597 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:07:11.168872 13223 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:07:12.980557 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:07:12.980737 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:07:17.983280 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:07:17.983448 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:07:22.884083 13276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:22.884289 13276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:07:22.884950 13276 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:07:22.886075 12207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:07:22.887214 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:22.887418 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:07:22.887526 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:07:22.887755 13276 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:07:22.984685 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:07:22.984884 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 14:07:27.299321 10767 heartbeater.cc:644] Failed to heartbeat to 127.9.224.189:42871 (613 consecutive failures): Network error: Failed to ping master at 127.9.224.189:42871: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:07:27.552839 11038 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (566 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:27.553050 10904 heartbeater.cc:644] Failed to heartbeat to 127.9.224.190:34475 (566 consecutive failures): Network error: Failed to ping master at 127.9.224.190:34475: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:07:27.986378 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:07:27.986547 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:07:28.722186 12212 maintenance_manager.cc:419] P 4163d103537745edb461e4dbdda724e6: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1451 bytes on disk
I20240501 14:07:28.727435 12211 maintenance_manager.cc:643] P 4163d103537745edb461e4dbdda724e6: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.005s sys 0.000s Metrics: {"dirs.queue_time_us":384,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":124,"thread_start_us":506,"threads_started":2,"wal-append.queue_time_us":313}
I20240501 14:07:29.028743 13311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:29.028952 13311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:07:29.029666 13311 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:07:29.030515 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:07:29.031939 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:29.032100 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:07:29.032219 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:07:29.032512 13311 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:07:32.987943 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:07:32.988111 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:07:33.557557 13332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:33.557765 13332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:07:33.558466 13332 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:07:33.560683 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:33.560954 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:07:33.561106 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:07:33.561360 13332 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:07:36.124593 13344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:36.124788 13344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:07:36.125490 13344 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:07:36.126523 12208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 14:07:36.127725 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:36.127924 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
I20240501 14:07:36.128029 12207 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:07:36.128253 13344 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:07:37.989787 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:07:37.989965 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 14:07:39.128926 13361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:39.129158 13361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47f54722b654655bfc871851b93abbc" member_type: VOTER last_known_addr { host: "127.9.224.190" port: 34475 } } peers { permanent_uuid: "e048fbcc4e544adea37b6bd9e5ac8543" member_type: VOTER last_known_addr { host: "127.9.224.189" port: 42871 } } peers { permanent_uuid: "4163d103537745edb461e4dbdda724e6" member_type: VOTER last_known_addr { host: "127.9.224.188" port: 38257 } }
I20240501 14:07:39.129884 13361 leader_election.cc:288] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e47f54722b654655bfc871851b93abbc (127.9.224.190:34475), e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871)
W20240501 14:07:39.132304 12207 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e048fbcc4e544adea37b6bd9e5ac8543 (127.9.224.189:42871): Network error: Client connection negotiation failed: client connection to 127.9.224.189:42871: connect: Connection refused (error 111)
W20240501 14:07:39.132517 12208 leader_election.cc:334] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e47f54722b654655bfc871851b93abbc (127.9.224.190:34475): Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
I20240501 14:07:39.132630 12208 leader_election.cc:302] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4163d103537745edb461e4dbdda724e6; no voters: e048fbcc4e544adea37b6bd9e5ac8543, e47f54722b654655bfc871851b93abbc
I20240501 14:07:39.132916 13361 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4163d103537745edb461e4dbdda724e6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 14:07:42.991544 12191 master_runner.cc:186] Error getting master registration for 127.9.224.190:34475: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111),
I20240501 14:07:42.991739 12191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 14:07:44.126875 11045 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:44.127198 11045 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:44.127295 11045 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
W20240501 14:07:44.127434 11045 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
F20240501 14:07:44.138561 11066 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1714572464 (unix time) try "date -d @1714572464" if you are using GNU date ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: T*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: T*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144996 11073 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144996 11073 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145047 11063 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e800002782) received by PID 10114 (TID 0x7f2f91a4e700) from PID 10114; stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144996 11073 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145047 11063 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145169 11068 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144996 11073 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145047 11063 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145169 11068 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145213 11057 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_351f54255b01483daad53a39989a377c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144996 11073 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145047 11063 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145169 11068 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145213 11057 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_351f54255b01483daad53a39989a377c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145455 11077 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144996 11073 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145047 11063 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145169 11068 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145213 11057 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_351f54255b01483daad53a39989a377c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145455 11077 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145787 11052 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a9db37a35094b309731f45b45ef8113 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f2fab31f980 (unknown) at ??:0
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144996 11073 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145047 11063 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145169 11068 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145213 11057 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_351f54255b01483daad53a39989a377c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145455 11077 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145787 11052 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a9db37a35094b309731f45b45ef8113 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.146987 11060 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f4862f63c8104085837aee11c33c2718 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.147022 1*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144996 11073 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145047 11063 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145169 11068 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145213 11057 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_351f54255b01483daad53a39989a377c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145455 11077 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145787 11052 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a9db37a35094b309731f45b45ef8113 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.146987 11060 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f4862f63c8104085837aee11c33c2718 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.147022 11059 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4801897b969f4fb8a4d46503057f4169 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 14:07:44.139552 11064 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.139763 11054 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a6090a9bb79448e7bd984f4fc6bb3217 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141052 11061 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a67258cb839442b69e96ce75fc4c281b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141192 11053 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e314decf1e7044ed9f9796b96a898931 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141238 11074 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141307 11058 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_557e6f54359d445eab8bcfb616e974ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141842 11070 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.141844 11056 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f616bd9bd35f4659896ecc727f466014 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.142972 11062 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143090 11069 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143198 11055 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_101396daa69f460997cbe7b34c1d12a7 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.143630 11078 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Fimed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111) 20240501 14:07:44.143630 11072 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144853 11065 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.144996 11073 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145047 11063 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145169 11068 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145213 11057 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_351f54255b01483daad53a39989a377c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145455 11077 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.145787 11052 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a9db37a35094b309731f45b45ef8113 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.146987 11060 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f4862f63c8104085837aee11c33c2718 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.147022 11059 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4801897b969f4fb8a4d46503057f4169 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)F20240501 14:07:44.147179 11067 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.9.224.190:34475,127.9.224.189:42871,127.9.224.188:38257, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.9.224.190:34475: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f2fa43cafb7 gsignal at ??:0
@ 0x7f2fa43cc921 abort at ??:0
@ 0x7f2fa70f3dcd google::LogMessage::Fail() at ??:0
@ 0x7f2fa70f7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f2fa70f37cc google::LogMessage::Flush() at ??:0
@ 0x7f2fa70f4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40a641 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:329
@ 0x44ab24 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9dd std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a99a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a94e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a689 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f2fa4a546df (unknown) at ??:0
@ 0x7f2fab3146db start_thread at ??:0
@ 0x7f2fa44ad71f clone at ??:0