Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240506 20:04:20.827948 23929 test_util.cc:268] Using random seed: -1825105123
2024-05-06T20:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T20:04:20Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3taq95/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3taq95/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/security.properties
2024-05-06 20:04:26: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3taq95/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3taq95/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20240506 20:04:33.419503 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.126:40141
--webserver_interface=127.23.94.126
--webserver_port=0
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:04:33.518294 24170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:33.518492 24170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:33.518543 24170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:33.521469 24170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:33.521517 24170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:33.521543 24170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:33.521566 24170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:33.524986 24170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.126:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.94.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:33.525735 24170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:33.526708 24170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:33.533368 24175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:33.533720 24178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:33.534135 24170 server_base.cc:1034] running on GCE node
W20240506 20:04:33.534422 24176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:33.534824 24170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:33.535585 24170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:33.536717 24170 hybrid_clock.cc:648] HybridClock initialized: now 1715025873536704 us; error 29 us; skew 500 ppm
I20240506 20:04:33.538290 24170 webserver.cc:460] Webserver started at http://127.23.94.126:45265/ using document root <none> and password file <none>
I20240506 20:04:33.538765 24170 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:33.538820 24170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:33.538973 24170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 20:04:33.540402 24170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/instance:
uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:33.540850 24170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal/instance:
uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:33.543723 24170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240506 20:04:33.545588 24184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:33.546432 24170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 20:04:33.546530 24170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:33.546599 24170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:33.562155 24170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:33.562636 24170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:33.562770 24170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:33.568902 24170 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.126:40141
I20240506 20:04:33.568907 24236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.126:40141 every 8 connection(s)
I20240506 20:04:33.569710 24170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/info.pb
I20240506 20:04:33.571925 24237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:33.573627 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 24170
I20240506 20:04:33.573748 23929 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal/instance
I20240506 20:04:33.574995 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--webserver_interface=127.23.94.125
--webserver_port=0
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240506 20:04:33.574934 24237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } has no permanent_uuid. Determining permanent_uuid...
I20240506 20:04:33.584484 24237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } has no permanent_uuid. Determining permanent_uuid...
W20240506 20:04:33.585163 24185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:33.585860 24237 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.94.125:42097: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:33.635061 24237 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } attempt: 1
W20240506 20:04:33.636358 24237 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.94.125:42097: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:33.677608 24238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:33.677775 24238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:33.677814 24238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:33.680780 24238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:33.680826 24238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:33.680845 24238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:33.680859 24238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:33.684263 24238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.94.125
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:33.685021 24238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:33.686045 24238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:33.692678 24249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:33.692997 24246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:33.693756 24247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:33.693789 24238 server_base.cc:1034] running on GCE node
I20240506 20:04:33.694233 24238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:33.695029 24238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:33.696180 24238 hybrid_clock.cc:648] HybridClock initialized: now 1715025873696165 us; error 27 us; skew 500 ppm
I20240506 20:04:33.697842 24238 webserver.cc:460] Webserver started at http://127.23.94.125:43623/ using document root <none> and password file <none>
I20240506 20:04:33.698323 24238 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:33.698376 24238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:33.698526 24238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 20:04:33.699981 24238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/instance:
uuid: "853e921441fd4ee4b5bb84fdab584029"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:33.700395 24238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal/instance:
uuid: "853e921441fd4ee4b5bb84fdab584029"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:33.703241 24238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240506 20:04:33.704531 24237 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } attempt: 2
I20240506 20:04:33.705185 24255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20240506 20:04:33.705705 24237 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.94.125:42097: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:33.706203 24238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:33.706301 24238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
uuid: "853e921441fd4ee4b5bb84fdab584029"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:33.706373 24238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:33.738847 24238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:33.739467 24238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:33.739610 24238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:33.745970 24238 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.125:42097
I20240506 20:04:33.745978 24307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.125:42097 every 8 connection(s)
I20240506 20:04:33.746843 24238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
I20240506 20:04:33.749250 24308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:33.750000 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 24238
I20240506 20:04:33.750101 23929 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal/instance
I20240506 20:04:33.750856 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--webserver_interface=127.23.94.124
--webserver_port=0
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240506 20:04:33.753021 24308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } has no permanent_uuid. Determining permanent_uuid...
I20240506 20:04:33.761533 24308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } has no permanent_uuid. Determining permanent_uuid...
I20240506 20:04:33.766582 24308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
W20240506 20:04:33.767170 24257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
W20240506 20:04:33.767867 24308 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.94.124:36421: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:33.796862 24237 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } attempt: 3
I20240506 20:04:33.800932 24237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
W20240506 20:04:33.802076 24237 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.94.124:36421: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:33.817041 24308 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } attempt: 1
W20240506 20:04:33.818037 24308 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.94.124:36421: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:33.833200 24237 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } attempt: 1
W20240506 20:04:33.834259 24237 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.94.124:36421: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
W20240506 20:04:33.853336 24309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:33.853523 24309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:33.853574 24309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:33.856459 24309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:33.856506 24309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:33.856531 24309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:33.856554 24309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:33.860021 24309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.94.124
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:33.860802 24309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:33.861754 24309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:33.868213 24318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:33.868580 24320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:33.868870 24317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:33.869354 24309 server_base.cc:1034] running on GCE node
I20240506 20:04:33.869829 24309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:33.870664 24309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:33.871811 24309 hybrid_clock.cc:648] HybridClock initialized: now 1715025873871796 us; error 27 us; skew 500 ppm
I20240506 20:04:33.873514 24309 webserver.cc:460] Webserver started at http://127.23.94.124:36977/ using document root <none> and password file <none>
I20240506 20:04:33.874020 24309 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:33.874075 24309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:33.874238 24309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 20:04:33.875811 24309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/instance:
uuid: "038cb801e9214358bab57093f322c097"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:33.876276 24309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal/instance:
uuid: "038cb801e9214358bab57093f322c097"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:33.879287 24309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240506 20:04:33.881228 24326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:33.882099 24309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:33.882198 24309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
uuid: "038cb801e9214358bab57093f322c097"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:33.882273 24309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:33.886179 24308 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } attempt: 2
W20240506 20:04:33.887408 24308 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.23.94.124:36421: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:33.897084 24309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:33.897611 24309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:33.897747 24309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:33.903856 24309 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.124:36421
I20240506 20:04:33.903867 24378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.124:36421 every 8 connection(s)
I20240506 20:04:33.904875 24309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
I20240506 20:04:33.905383 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 24309
I20240506 20:04:33.905483 23929 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal/instance
I20240506 20:04:33.907758 24379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:33.909395 24237 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } attempt: 2
I20240506 20:04:33.911854 24379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } has no permanent_uuid. Determining permanent_uuid...
I20240506 20:04:33.918592 24379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } has no permanent_uuid. Determining permanent_uuid...
I20240506 20:04:33.922197 24237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Bootstrap starting.
I20240506 20:04:33.922528 24379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
I20240506 20:04:33.925570 24237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:33.926934 24237 log.cc:824] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:33.927554 24379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap starting.
I20240506 20:04:33.928511 24237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: No bootstrap required, opened a new log
I20240506 20:04:33.929625 24379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:33.930357 24379 log.cc:824] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:33.931625 24237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:33.931793 24237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:33.931854 24237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25192b33b13d4b089dc3fb1d1f6fc2e9, State: Initialized, Role: FOLLOWER
I20240506 20:04:33.932125 24379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: No bootstrap required, opened a new log
I20240506 20:04:33.932296 24237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:33.932718 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:33.932818 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:33.933378 24237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:04:33.935627 24379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:33.935799 24379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:33.935853 24379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038cb801e9214358bab57093f322c097, State: Initialized, Role: FOLLOWER
I20240506 20:04:33.936228 24379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:33.936664 24403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:33.936797 24403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:33.937256 24379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: configured and running, proceeding with master startup.
W20240506 20:04:33.943313 24417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 20:04:33.943425 24417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 20:04:33.942268 24170 master_runner.cc:186] Error getting master registration for 127.23.94.125:42097: OK, instance_id { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" instance_seqno: 1715025873744317 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240506 20:04:33.944718 24170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:33.944464 24309 master_runner.cc:186] Error getting master registration for 127.23.94.125:42097: OK, instance_id { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" instance_seqno: 1715025873744317 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240506 20:04:33.946749 24309 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:04:33.947474 24402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 20:04:33.947530 24402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 20:04:33.978628 24308 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } attempt: 3
I20240506 20:04:33.985939 24308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap starting.
I20240506 20:04:33.988390 24308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:33.989696 24308 log.cc:824] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:33.993307 24308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: No bootstrap required, opened a new log
I20240506 20:04:33.996711 24308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:33.996937 24308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:33.997020 24308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853e921441fd4ee4b5bb84fdab584029, State: Initialized, Role: FOLLOWER
I20240506 20:04:33.997531 24308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:33.998728 24308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:04:34.009338 24238 master_runner.cc:263] No leader master found from master 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:34.009406 24238 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:04:34.012425 24437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240506 20:04:34.012588 24437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240506 20:04:34.014081 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:34.014259 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:34.048141 24388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:34.048345 24388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:34.049374 24388 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:34.050055 24282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029" is_pre_election: true
I20240506 20:04:34.050338 24282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 0.
I20240506 20:04:34.050804 24353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097" is_pre_election: true
I20240506 20:04:34.050866 24185 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029; no voters:
I20240506 20:04:34.051065 24353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 0.
I20240506 20:04:34.051229 24388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 20:04:34.051293 24388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 20:04:34.051333 24388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:34.052404 24388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:34.052755 24388 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 1 election: Requested vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:34.053124 24353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097"
I20240506 20:04:34.053218 24353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:34.054181 24353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 1.
I20240506 20:04:34.054452 24186 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters:
I20240506 20:04:34.055018 24282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029"
I20240506 20:04:34.055125 24282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:34.056671 24282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 1.
I20240506 20:04:34.057283 24388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 20:04:34.057461 24388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 1 LEADER]: Becoming Leader. State: Replica: 25192b33b13d4b089dc3fb1d1f6fc2e9, State: Running, Role: LEADER
I20240506 20:04:34.057777 24388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:34.059571 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25192b33b13d4b089dc3fb1d1f6fc2e9. Latest consensus state: current_term: 1 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:34.059744 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: LEADER
W20240506 20:04:34.064147 24190 tablet.cc:2369] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 20:04:34.074237 24442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:04:34.075314 24442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:04:34.092046 24353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 1 FOLLOWER]: Refusing update from remote peer 25192b33b13d4b089dc3fb1d1f6fc2e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 20:04:34.092859 24440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 20:04:34.096935 24403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25192b33b13d4b089dc3fb1d1f6fc2e9. Latest consensus state: current_term: 1 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:34.097050 24403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:34.098028 24282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 1 FOLLOWER]: Refusing update from remote peer 25192b33b13d4b089dc3fb1d1f6fc2e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 20:04:34.098618 24440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 20:04:34.102020 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25192b33b13d4b089dc3fb1d1f6fc2e9. Latest consensus state: current_term: 1 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:34.102126 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:34.106540 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:34.106664 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:34.106997 24443 mvcc.cc:205] Tried to move back new op lower bound from 7024745980273709056 to 7024745980144119808. Current Snapshot: MvccSnapshot[applied={T|T < 7024745980273709056}]
I20240506 20:04:34.106992 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:34.107100 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:34.109993 24440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:34.110097 24440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:34.111045 24403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:34.111152 24403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:34.111621 24446 mvcc.cc:205] Tried to move back new op lower bound from 7024745980273709056 to 7024745980144119808. Current Snapshot: MvccSnapshot[applied={T|T < 7024745980273709056}]
I20240506 20:04:34.112871 24444 mvcc.cc:205] Tried to move back new op lower bound from 7024745980273709056 to 7024745980144119808. Current Snapshot: MvccSnapshot[applied={T|T < 7024745980273709056}]
I20240506 20:04:34.118072 24442 catalog_manager.cc:1347] Generated new cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:34.118134 24442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:04:34.158035 24442 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 20:04:34.159250 24442 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:04:34.175269 24442 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Generated new TSK 0
I20240506 20:04:34.176249 24442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:04:34.176996 24442 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:04:34.177542 24442 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240506 20:04:34.203104 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.94.65:0
--local_ip_for_outbound_sockets=127.23.94.65
--webserver_interface=127.23.94.65
--webserver_port=0
--tserver_master_addrs=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20240506 20:04:34.309646 24452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:34.309906 24452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:34.309976 24452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:34.313068 24452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:34.313197 24452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.94.65
I20240506 20:04:34.316856 24452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.94.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.94.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--never_fsync=true
--heap_profile_path=/tmp/kudu.24452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.94.65
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:34.317725 24452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:34.318787 24452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:34.326853 24457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:34.326987 24458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:34.327374 24460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:34.327804 24452 server_base.cc:1034] running on GCE node
I20240506 20:04:34.328133 24452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:34.328725 24452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:34.329870 24452 hybrid_clock.cc:648] HybridClock initialized: now 1715025874329862 us; error 28 us; skew 500 ppm
I20240506 20:04:34.331485 24452 webserver.cc:460] Webserver started at http://127.23.94.65:42593/ using document root <none> and password file <none>
I20240506 20:04:34.331988 24452 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:34.332043 24452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:34.332211 24452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 20:04:34.333914 24452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/data/instance:
uuid: "3ec4b56ff88545a69ab015a9d695d5d9"
format_stamp: "Formatted at 2024-05-06 20:04:34 on dist-test-slave-8c1w"
I20240506 20:04:34.334442 24452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec4b56ff88545a69ab015a9d695d5d9"
format_stamp: "Formatted at 2024-05-06 20:04:34 on dist-test-slave-8c1w"
I20240506 20:04:34.337617 24452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240506 20:04:34.339699 24466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:34.340665 24452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 20:04:34.340763 24452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal
uuid: "3ec4b56ff88545a69ab015a9d695d5d9"
format_stamp: "Formatted at 2024-05-06 20:04:34 on dist-test-slave-8c1w"
I20240506 20:04:34.340842 24452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:34.357549 24452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:34.358103 24452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:34.358255 24452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:34.358763 24452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 20:04:34.359645 24452 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 20:04:34.359700 24452 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:34.359755 24452 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 20:04:34.359779 24452 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:34.367369 24452 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.65:36667
I20240506 20:04:34.367414 24579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.65:36667 every 8 connection(s)
I20240506 20:04:34.368207 24452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/data/info.pb
I20240506 20:04:34.376865 24582 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:34.376909 24581 heartbeater.cc:347] Connected to a master server at 127.23.94.126:40141
I20240506 20:04:34.376865 24580 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:34.377107 24580 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:34.377106 24581 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:34.377106 24582 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:34.377283 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 24452
I20240506 20:04:34.377386 23929 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal/instance
I20240506 20:04:34.377697 24580 heartbeater.cc:511] Master 127.23.94.124:36421 requested a full tablet report, sending...
I20240506 20:04:34.377697 24582 heartbeater.cc:511] Master 127.23.94.125:42097 requested a full tablet report, sending...
I20240506 20:04:34.378269 24581 heartbeater.cc:511] Master 127.23.94.126:40141 requested a full tablet report, sending...
I20240506 20:04:34.379052 24272 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:34.379494 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.94.66:0
--local_ip_for_outbound_sockets=127.23.94.66
--webserver_interface=127.23.94.66
--webserver_port=0
--tserver_master_addrs=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240506 20:04:34.380043 24343 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:34.380136 24200 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:34.381337 24200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.94.65:53247
W20240506 20:04:34.484377 24587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:34.484560 24587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:34.484604 24587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:34.487576 24587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:34.487673 24587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.94.66
I20240506 20:04:34.491081 24587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.94.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.94.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--never_fsync=true
--heap_profile_path=/tmp/kudu.24587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.94.66
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:34.491786 24587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:34.492750 24587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:34.500429 24592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:34.500574 24595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:34.500909 24593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:34.501312 24587 server_base.cc:1034] running on GCE node
I20240506 20:04:34.501633 24587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:34.502125 24587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:34.503273 24587 hybrid_clock.cc:648] HybridClock initialized: now 1715025874503256 us; error 34 us; skew 500 ppm
I20240506 20:04:34.504792 24587 webserver.cc:460] Webserver started at http://127.23.94.66:40445/ using document root <none> and password file <none>
I20240506 20:04:34.505291 24587 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:34.505342 24587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:34.505493 24587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 20:04:34.506918 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/data/instance:
uuid: "c35a1e4b657846ba945aa2602bea72cb"
format_stamp: "Formatted at 2024-05-06 20:04:34 on dist-test-slave-8c1w"
I20240506 20:04:34.507337 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal/instance:
uuid: "c35a1e4b657846ba945aa2602bea72cb"
format_stamp: "Formatted at 2024-05-06 20:04:34 on dist-test-slave-8c1w"
I20240506 20:04:34.510382 24587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240506 20:04:34.512254 24601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:34.513126 24587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:34.513212 24587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal
uuid: "c35a1e4b657846ba945aa2602bea72cb"
format_stamp: "Formatted at 2024-05-06 20:04:34 on dist-test-slave-8c1w"
I20240506 20:04:34.513286 24587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:34.528156 24587 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:34.528606 24587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:34.528761 24587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:34.529196 24587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 20:04:34.529997 24587 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 20:04:34.530043 24587 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:34.530081 24587 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 20:04:34.530107 24587 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:34.538247 24587 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.66:35561
I20240506 20:04:34.538270 24714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.66:35561 every 8 connection(s)
I20240506 20:04:34.539130 24587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/data/info.pb
I20240506 20:04:34.545776 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 24587
I20240506 20:04:34.545872 23929 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal/instance
I20240506 20:04:34.548058 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.94.67:0
--local_ip_for_outbound_sockets=127.23.94.67
--webserver_interface=127.23.94.67
--webserver_port=0
--tserver_master_addrs=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240506 20:04:34.548280 24717 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:34.548470 24717 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:34.548535 24715 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:34.548586 24715 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:34.549074 24717 heartbeater.cc:511] Master 127.23.94.125:42097 requested a full tablet report, sending...
I20240506 20:04:34.549346 24716 heartbeater.cc:347] Connected to a master server at 127.23.94.126:40141
I20240506 20:04:34.549405 24716 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:34.549562 24716 heartbeater.cc:511] Master 127.23.94.126:40141 requested a full tablet report, sending...
I20240506 20:04:34.549074 24715 heartbeater.cc:511] Master 127.23.94.124:36421 requested a full tablet report, sending...
I20240506 20:04:34.550087 24272 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:34.550158 24200 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:34.550745 24200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.94.66:59393
I20240506 20:04:34.551163 24343 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:04:34.654322 24722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:34.654517 24722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:34.654573 24722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:34.657526 24722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:34.657630 24722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.94.67
I20240506 20:04:34.661018 24722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.94.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.94.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--never_fsync=true
--heap_profile_path=/tmp/kudu.24722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.94.67
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:34.661751 24722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:34.662709 24722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:34.670020 24727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:34.670327 24730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:34.670698 24728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:34.671111 24722 server_base.cc:1034] running on GCE node
I20240506 20:04:34.671415 24722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:34.671917 24722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:34.673048 24722 hybrid_clock.cc:648] HybridClock initialized: now 1715025874673029 us; error 32 us; skew 500 ppm
I20240506 20:04:34.674628 24722 webserver.cc:460] Webserver started at http://127.23.94.67:43609/ using document root <none> and password file <none>
I20240506 20:04:34.674945 24735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:34.675048 24735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:34.675189 24722 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:34.675283 24722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:34.675285 24736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:34.675403 24736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:34.675482 24722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 20:04:34.676105 24735 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:34.676457 24282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029" is_pre_election: true
I20240506 20:04:34.676580 24282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038cb801e9214358bab57093f322c097 in term 1.
I20240506 20:04:34.676527 24211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" is_pre_election: true
I20240506 20:04:34.676529 24210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" is_pre_election: true
I20240506 20:04:34.676971 24327 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029; no voters:
I20240506 20:04:34.676977 24353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097" is_pre_election: true
I20240506 20:04:34.677141 24353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853e921441fd4ee4b5bb84fdab584029 in term 1.
I20240506 20:04:34.677237 24736 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
I20240506 20:04:34.677385 24737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 20:04:34.677386 24257 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:34.677438 24737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:34.677495 24737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 1 FOLLOWER]: Advancing to term 2
I20240506 20:04:34.677702 24735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 20:04:34.677752 24735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:34.677788 24735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 1 FOLLOWER]: Advancing to term 2
I20240506 20:04:34.678653 24737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:34.678889 24735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:34.679194 24282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029"
I20240506 20:04:34.679194 24210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
I20240506 20:04:34.679324 24282 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 038cb801e9214358bab57093f322c097 in current term 2: Already voted for candidate 853e921441fd4ee4b5bb84fdab584029 in this term.
I20240506 20:04:34.679576 24327 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:34.679663 24210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
I20240506 20:04:34.679536 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/data/instance:
uuid: "b0ecd08373ba4b2cb8623b3f4862bb98"
format_stamp: "Formatted at 2024-05-06 20:04:34 on dist-test-slave-8c1w"
I20240506 20:04:34.679806 24737 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 2 election: Requested vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
I20240506 20:04:34.679941 24737 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 20:04:34.679958 24353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097"
I20240506 20:04:34.680102 24353 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 853e921441fd4ee4b5bb84fdab584029 in current term 2: Already voted for candidate 038cb801e9214358bab57093f322c097 in this term.
I20240506 20:04:34.680089 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal/instance:
uuid: "b0ecd08373ba4b2cb8623b3f4862bb98"
format_stamp: "Formatted at 2024-05-06 20:04:34 on dist-test-slave-8c1w"
I20240506 20:04:34.680347 24735 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 2 election: Requested vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:34.680390 24257 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:34.680529 24735 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 20:04:34.683082 24722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240506 20:04:34.684957 24739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:34.685838 24722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:34.685926 24722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal
uuid: "b0ecd08373ba4b2cb8623b3f4862bb98"
format_stamp: "Formatted at 2024-05-06 20:04:34 on dist-test-slave-8c1w"
I20240506 20:04:34.686002 24722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:34.699417 24722 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:34.699932 24722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:34.700064 24722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:34.700573 24722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 20:04:34.701423 24722 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 20:04:34.701475 24722 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:34.701522 24722 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 20:04:34.701546 24722 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:34.709271 24722 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.67:44791
I20240506 20:04:34.709317 24852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.67:44791 every 8 connection(s)
I20240506 20:04:34.710047 24722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/data/info.pb
I20240506 20:04:34.714619 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 24722
I20240506 20:04:34.714713 23929 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal/instance
I20240506 20:04:34.718665 24853 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:34.718858 24853 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:34.719166 24855 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:34.719235 24855 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:34.719329 24853 heartbeater.cc:511] Master 127.23.94.124:36421 requested a full tablet report, sending...
I20240506 20:04:34.719396 24855 heartbeater.cc:511] Master 127.23.94.125:42097 requested a full tablet report, sending...
I20240506 20:04:34.720016 24854 heartbeater.cc:347] Connected to a master server at 127.23.94.126:40141
I20240506 20:04:34.720093 24854 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:34.720256 24854 heartbeater.cc:511] Master 127.23.94.126:40141 requested a full tablet report, sending...
I20240506 20:04:34.720292 24272 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:34.720827 24343 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:34.721163 24200 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:34.721627 24200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.94.67:50661
I20240506 20:04:34.733328 23929 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 20:04:34.745143 23929 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3taq95/build/debug/bin/kudu with pid 24238
I20240506 20:04:34.746416 24200 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_2ee32586537f466fa07ee5ad6c48dec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.746826 24201 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_7f7588f0e29e4057a66f08687ec28975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.747085 24199 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_a1b4d3a0d5ce479584c8f895844cadc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.747318 24198 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_e80158bfa50b427cbd790f160164b525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.747531 24197 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_c66444cb675241d1a23c4348383b208e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.752740 24196 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_638fa9e2f7b349a38f86c3d585998fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.752918 24195 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_c568fe36c59c4d93bda72f412845b2f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.753185 24194 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_6d90713e02f146279b23811c97941079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.753422 24193 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_8df5615aacee4940a5402206a921f0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.754410 24192 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_758d3e4f504a489eb12b7bf93d583cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:34.765245 24353 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 2 FOLLOWER]: Rejecting Update request from peer 25192b33b13d4b089dc3fb1d1f6fc2e9 for earlier term 1. Current term is 2. Ops: []
I20240506 20:04:34.765681 24899 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 20:04:34.765924 24899 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 20:04:34.765985 24899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 25192b33b13d4b089dc3fb1d1f6fc2e9, State: Running, Role: LEADER
I20240506 20:04:34.766191 24899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:34.766475 24899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 1 FOLLOWER]: Advancing to term 2
W20240506 20:04:34.781710 24199 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a1b4d3a0d5ce479584c8f895844cadc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.781807 24196 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_638fa9e2f7b349a38f86c3d585998fea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.781828 24197 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c66444cb675241d1a23c4348383b208e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.782063 24198 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e80158bfa50b427cbd790f160164b525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.782217 24193 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8df5615aacee4940a5402206a921f0ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.782339 24201 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7f7588f0e29e4057a66f08687ec28975 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.782465 24194 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6d90713e02f146279b23811c97941079 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.782900 24195 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c568fe36c59c4d93bda72f412845b2f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.783607 24200 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2ee32586537f466fa07ee5ad6c48dec6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.781711 24192 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_758d3e4f504a489eb12b7bf93d583cb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240506 20:04:34.882725 24582 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:34.883220 24581 heartbeater.cc:503] Master 127.23.94.126:40141 was elected leader, sending a full tablet report...
I20240506 20:04:34.944981 24417 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:34.945091 24417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: loading cluster ID for follower catalog manager: success
I20240506 20:04:34.947530 24417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: acquiring CA information for follower catalog manager: success
I20240506 20:04:34.948969 24402 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:04:34.949215 24417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240506 20:04:35.051955 24717 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:35.052250 24716 heartbeater.cc:503] Master 127.23.94.126:40141 was elected leader, sending a full tablet report...
I20240506 20:04:35.137917 24197 catalog_manager.cc:2145] created HMS catalog entry for table [id=8d7f9026d6a34a7da7d193171e1e33bb]
W20240506 20:04:35.138880 24197 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
I20240506 20:04:35.163072 24195 catalog_manager.cc:2145] created HMS catalog entry for table [id=f8c2559276c74cc698dc2e82a2d7bf5f]
W20240506 20:04:35.163614 24195 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
I20240506 20:04:35.185608 24198 catalog_manager.cc:2145] created HMS catalog entry for table [id=906d0bea77ee4a8f9a2c24d07ff40655]
W20240506 20:04:35.186120 24198 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
I20240506 20:04:35.205989 24201 catalog_manager.cc:2145] created HMS catalog entry for table [id=83e3d80488894483b90b51b8631ea18e]
W20240506 20:04:35.206465 24201 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
W20240506 20:04:35.222715 24855 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:35.222913 24854 heartbeater.cc:503] Master 127.23.94.126:40141 was elected leader, sending a full tablet report...
I20240506 20:04:35.227324 24193 catalog_manager.cc:2145] created HMS catalog entry for table [id=e33e55f6620e4d90bfd0092c65c2aecd]
W20240506 20:04:35.227771 24193 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
I20240506 20:04:35.262039 24194 catalog_manager.cc:2145] created HMS catalog entry for table [id=4b49fca39a0a4f789d4cd4f791063786]
W20240506 20:04:35.262564 24194 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
I20240506 20:04:35.284907 24200 catalog_manager.cc:2145] created HMS catalog entry for table [id=eb6dd8053f18430da37b1eb0a3931948]
W20240506 20:04:35.285522 24200 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
I20240506 20:04:35.306869 24196 catalog_manager.cc:2145] created HMS catalog entry for table [id=fced752d1b68486d8b91c515f1181350]
W20240506 20:04:35.307410 24196 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
I20240506 20:04:35.314617 24904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:35.314760 24904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:35.315680 24210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" is_pre_election: true
I20240506 20:04:35.315874 24210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038cb801e9214358bab57093f322c097 in term 2.
I20240506 20:04:35.316308 24328 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters:
I20240506 20:04:35.316607 24905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 20:04:35.316699 24905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 20:04:35.316751 24905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 2 FOLLOWER]: Advancing to term 3
I20240506 20:04:35.317225 24904 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:35.317610 24327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:35.318336 24905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:35.318924 24905 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 3 election: Requested vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
I20240506 20:04:35.319314 24210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
I20240506 20:04:35.319417 24210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 2 FOLLOWER]: Advancing to term 3
W20240506 20:04:35.320014 24327 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:35.320431 24210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038cb801e9214358bab57093f322c097 in term 3.
I20240506 20:04:35.320753 24328 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters:
W20240506 20:04:35.320828 24327 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:35.321002 24905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 20:04:35.321132 24905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 3 LEADER]: Becoming Leader. State: Replica: 038cb801e9214358bab57093f322c097, State: Running, Role: LEADER
I20240506 20:04:35.321427 24905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:35.322383 24904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 038cb801e9214358bab57093f322c097. Latest consensus state: current_term: 3 leader_uuid: "038cb801e9214358bab57093f322c097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:35.322484 24904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:35.322767 24909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:04:35.323536 24909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:04:35.323874 24909 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:35.323918 24909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:04:35.324429 24909 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:04:35.324827 24909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Loaded TSK: 0
I20240506 20:04:35.325526 24909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:04:35.325901 24909 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:04:35.326210 24909 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240506 20:04:35.333263 24192 catalog_manager.cc:2145] created HMS catalog entry for table [id=4935d9be7af749d8863a707db7336e33]
W20240506 20:04:35.333882 24192 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
I20240506 20:04:35.352682 24199 catalog_manager.cc:2145] created HMS catalog entry for table [id=59d2865ad3bc4f8da8ba8011111472c7]
W20240506 20:04:35.353252 24199 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER
I20240506 20:04:35.657020 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--webserver_interface=127.23.94.125
--webserver_port=43623
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:04:35.716604 24327 consensus_peers.cc:485] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 -> Peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Couldn't send request to peer 853e921441fd4ee4b5bb84fdab584029. Status: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240506 20:04:35.807202 24912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:35.807570 24912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:35.807686 24912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:35.812058 24912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:35.812263 24912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:35.812355 24912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:35.812434 24912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:35.817008 24912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.94.125
--webserver_port=43623
--never_fsync=true
--heap_profile_path=/tmp/kudu.24912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:35.817873 24912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:35.818958 24912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:35.828307 24919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:35.828702 24921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:35.828927 24918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:35.829386 24912 server_base.cc:1034] running on GCE node
I20240506 20:04:35.829955 24912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:35.831033 24912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:35.832206 24912 hybrid_clock.cc:648] HybridClock initialized: now 1715025875832182 us; error 46 us; skew 500 ppm
I20240506 20:04:35.834795 24912 webserver.cc:460] Webserver started at http://127.23.94.125:43623/ using document root <none> and password file <none>
I20240506 20:04:35.835543 24912 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:35.835635 24912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:35.840095 24912 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240506 20:04:35.842530 24927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:35.843781 24912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:35.843914 24912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
uuid: "853e921441fd4ee4b5bb84fdab584029"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:35.844539 24912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:35.863894 24210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 3 FOLLOWER]: Refusing update from remote peer 038cb801e9214358bab57093f322c097: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240506 20:04:35.864614 24905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240506 20:04:35.865792 24912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:35.866564 24912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:35.866768 24912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:35.867282 24905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "038cb801e9214358bab57093f322c097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:35.867389 24905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:35.869012 24940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 038cb801e9214358bab57093f322c097. Latest consensus state: current_term: 3 leader_uuid: "038cb801e9214358bab57093f322c097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:35.869225 24940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:35.869357 24940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "038cb801e9214358bab57093f322c097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:35.869580 24940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:35.874922 24197 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=8d7f9026d6a34a7da7d193171e1e33bb]
I20240506 20:04:35.875362 24197 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 5) took 1129 ms. Trace:
I20240506 20:04:35.876045 24912 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.125:42097
I20240506 20:04:35.875440 24197 rpcz_store.cc:276] 0506 20:04:34.746063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.747454 (+ 1391us) service_pool.cc:225] Handling call
0506 20:04:34.782732 (+ 35278us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.782975 (+ 243us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.137905 (+354930us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.138786 (+ 881us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.138789 (+ 3us) write_op.cc:454] Released schema lock
0506 20:04:35.138953 (+ 164us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:35.875343 (+736390us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.138716 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.138727 (+ 11us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.138732 (+ 5us) write_op.cc:454] Released schema lock
0506 20:04:35.138735 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":213604,"Hive Metastore.run_cpu_time_us":666,"Hive Metastore.run_wall_time_us":876832,"child_traces":[["op",{}]]}
W20240506 20:04:35.876555 24197 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.23.94.65:53247 (request call id 2) took 992 ms (client timeout 1000 ms). Trace:
W20240506 20:04:35.876638 24197 rpcz_store.cc:269] 0506 20:04:34.883631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:35.876249 (+992618us) service_pool.cc:225] Handling call
0506 20:04:35.876535 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240506 20:04:35.876964 24197 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.23.94.66:59393 (request call id 2) took 824 ms (client timeout 1000 ms). Trace:
W20240506 20:04:35.877024 24197 rpcz_store.cc:269] 0506 20:04:35.052673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:35.876774 (+824101us) service_pool.cc:225] Handling call
0506 20:04:35.876954 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 20:04:35.877820 24912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
W20240506 20:04:35.878438 24861 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.94.126:40141)
I20240506 20:04:35.882227 24343 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60418:
name: "default.Table_c66444cb675241d1a23c4348383b208e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:35.883894 24987 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:04:35.884802 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 24912
I20240506 20:04:35.885488 24580 heartbeater.cc:503] Master 127.23.94.124:36421 was elected leader, sending a full tablet report...
I20240506 20:04:35.885483 24986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.125:42097 every 8 connection(s)
I20240506 20:04:35.888883 24987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap starting.
I20240506 20:04:35.895076 24987 log.cc:824] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:35.900390 24416 catalog_manager.cc:2706] Deleting table default.table_c66444cb675241d1a23c4348383b208e [id=8d7f9026d6a34a7da7d193171e1e33bb] in response to Hive Metastore notification log event 11
W20240506 20:04:35.900774 24416 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 11 DROP_TABLE default.table_c66444cb675241d1a23c4348383b208e: Not found: the table does not exist: table_id: "8d7f9026d6a34a7da7d193171e1e33bb" table_name: "default.table_c66444cb675241d1a23c4348383b208e"
I20240506 20:04:35.901587 24987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 20:04:35.901939 24987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap complete.
I20240506 20:04:35.905928 24987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:35.906425 24987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853e921441fd4ee4b5bb84fdab584029, State: Initialized, Role: FOLLOWER
I20240506 20:04:35.906980 24987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:35.908192 24987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: configured and running, proceeding with master startup.
W20240506 20:04:35.912282 24343 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c66444cb675241d1a23c4348383b208e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 20:04:35.913604 24997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:35.913728 24997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:35.916946 25011 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:35.917006 25011 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: loading cluster ID for follower catalog manager: success
I20240506 20:04:35.919214 25011 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: acquiring CA information for follower catalog manager: success
I20240506 20:04:35.921829 25011 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:04:35.924065 24582 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:35.925273 24951 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:35139; Asking this server to re-register.
I20240506 20:04:35.925941 24582 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:35.927026 24951 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:36.024808 24343 catalog_manager.cc:2145] created HMS catalog entry for table [id=ee3b4ffa4c254f02b890d1ac3b7261da]
I20240506 20:04:36.033776 24343 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60418:
table { table_name: "default.Table_c66444cb675241d1a23c4348383b208e" } new_table_name: "default.Table_ea33dccc2852419e87b57fc573f6572e" modify_external_catalogs: true
I20240506 20:04:36.042327 24416 catalog_manager.cc:2706] Deleting table default.table_c568fe36c59c4d93bda72f412845b2f2 [id=f8c2559276c74cc698dc2e82a2d7bf5f] in response to Hive Metastore notification log event 12
W20240506 20:04:36.042505 24416 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_c568fe36c59c4d93bda72f412845b2f2: Not found: the table does not exist: table_id: "f8c2559276c74cc698dc2e82a2d7bf5f" table_name: "default.table_c568fe36c59c4d93bda72f412845b2f2"
F20240506 20:04:36.042557 24416 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 12): 12 CREATE_TABLE default.table_c66444cb675241d1a23c4348383b208e
*** Check failure stack trace: ***
I20240506 20:04:36.048029 24195 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=f8c2559276c74cc698dc2e82a2d7bf5f]
I20240506 20:04:36.048243 24195 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 7) took 1295 ms. Trace:
I20240506 20:04:36.048295 24195 rpcz_store.cc:276] 0506 20:04:34.752653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.752805 (+ 152us) service_pool.cc:225] Handling call
0506 20:04:34.782964 (+ 30159us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.783226 (+ 262us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.163063 (+379837us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.163562 (+ 499us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.163564 (+ 2us) write_op.cc:454] Released schema lock
0506 20:04:35.163694 (+ 130us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:36.048230 (+884536us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.163514 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.163521 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.163526 (+ 5us) write_op.cc:454] Released schema lock
0506 20:04:35.163529 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1065671,"Hive Metastore.run_cpu_time_us":377,"Hive Metastore.run_wall_time_us":198050,"child_traces":[["op",{}]]}
I20240506 20:04:36.051806 24342 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60418:
name: "default.Table_c568fe36c59c4d93bda72f412845b2f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:36.054106 24715 heartbeater.cc:503] Master 127.23.94.124:36421 was elected leader, sending a full tablet report...
I20240506 20:04:36.057235 24649 tablet_service.cc:1467] Processing CreateTablet for tablet 04c1b96ddace45fe8c59d8d38437b0cb (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:36.058231 24649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c1b96ddace45fe8c59d8d38437b0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:36.068811 24514 tablet_service.cc:1467] Processing CreateTablet for tablet 04c1b96ddace45fe8c59d8d38437b0cb (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:36.069766 24514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c1b96ddace45fe8c59d8d38437b0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:36.070228 25024 tablet_bootstrap.cc:492] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: Bootstrap starting.
I20240506 20:04:36.070704 24717 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:36.072005 24951 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:52841; Asking this server to re-register.
I20240506 20:04:36.072463 24717 heartbeater.cc:464] Registering TS with master...
@ 0x7f5e1feb3dcd google::LogMessage::Fail() at ??:0
@ 0x7f5e1feb7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5e1feb37cc google::LogMessage::Flush() at ??:0
I20240506 20:04:36.077227 24951 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:36.078787 25024 tablet_bootstrap.cc:654] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: Neither blocks nor log segments found. Creating new log.
@ 0x7f5e1feb3a11 google::LogMessage::~LogMessage() at ??:0
I20240506 20:04:36.079588 25024 log.cc:824] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:36.081175 24787 tablet_service.cc:1467] Processing CreateTablet for tablet 04c1b96ddace45fe8c59d8d38437b0cb (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:36.082089 24787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c1b96ddace45fe8c59d8d38437b0cb. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f5e24ddf4f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240506 20:04:36.085618 25024 tablet_bootstrap.cc:492] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: No bootstrap required, opened a new log
I20240506 20:04:36.085790 25024 ts_tablet_manager.cc:1367] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240506 20:04:36.088948 25024 raft_consensus.cc:348] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.089152 25024 raft_consensus.cc:374] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:36.089560 25028 tablet_bootstrap.cc:492] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: Bootstrap starting.
I20240506 20:04:36.090427 25024 raft_consensus.cc:729] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35a1e4b657846ba945aa2602bea72cb, State: Initialized, Role: FOLLOWER
I20240506 20:04:36.090868 25024 consensus_queue.cc:260] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
@ 0x7f5e24dde5ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240506 20:04:36.091607 24341 master_service.cc:946] ReplaceTablet: received request to replace tablet 04c1b96ddace45fe8c59d8d38437b0cb from {username='slave'} at 127.0.0.1:60418
I20240506 20:04:36.091763 25024 ts_tablet_manager.cc:1398] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240506 20:04:36.092574 24340 master_service.cc:946] ReplaceTablet: received request to replace tablet 04c1b96ddace45fe8c59d8d38437b0cb from {username='slave'} at 127.0.0.1:60418
I20240506 20:04:36.092757 24339 master_service.cc:946] ReplaceTablet: received request to replace tablet 04c1b96ddace45fe8c59d8d38437b0cb from {username='slave'} at 127.0.0.1:60418
I20240506 20:04:36.092888 24338 master_service.cc:946] ReplaceTablet: received request to replace tablet 04c1b96ddace45fe8c59d8d38437b0cb from {username='slave'} at 127.0.0.1:60418
I20240506 20:04:36.093029 24337 master_service.cc:946] ReplaceTablet: received request to replace tablet 04c1b96ddace45fe8c59d8d38437b0cb from {username='slave'} at 127.0.0.1:60418
@ 0x7f5e24dddd51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20240506 20:04:36.098209 24341 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Sending DeleteTablet for 3 replicas of tablet 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:36.100037 24341 catalog_manager.cc:6554] ReplaceTablet: tablet 04c1b96ddace45fe8c59d8d38437b0cb deleted and replaced by tablet 7431e9db4eca4dda856cece3204e666a
I20240506 20:04:36.101222 24648 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c1b96ddace45fe8c59d8d38437b0cb with delete_type TABLET_DATA_DELETED (replaced by tablet 7431e9db4eca4dda856cece3204e666a) from {username='slave'} at 127.0.0.1:42036
I20240506 20:04:36.101325 24514 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c1b96ddace45fe8c59d8d38437b0cb with delete_type TABLET_DATA_DELETED (replaced by tablet 7431e9db4eca4dda856cece3204e666a) from {username='slave'} at 127.0.0.1:54332
I20240506 20:04:36.105525 24786 tablet_service.cc:1467] Processing CreateTablet for tablet 7431e9db4eca4dda856cece3204e666a (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:36.105614 25030 tablet_replica.cc:330] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: stopping tablet replica
I20240506 20:04:36.105792 24786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7431e9db4eca4dda856cece3204e666a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:36.105832 24514 tablet_service.cc:1467] Processing CreateTablet for tablet 7431e9db4eca4dda856cece3204e666a (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:36.105876 25030 raft_consensus.cc:2229] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 20:04:36.105922 24648 tablet_service.cc:1467] Processing CreateTablet for tablet 7431e9db4eca4dda856cece3204e666a (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:36.106071 24514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7431e9db4eca4dda856cece3204e666a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:36.106112 25030 raft_consensus.cc:2258] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:36.106154 24648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7431e9db4eca4dda856cece3204e666a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:36.108526 25024 tablet_bootstrap.cc:492] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: Bootstrap starting.
I20240506 20:04:36.109138 24649 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c1b96ddace45fe8c59d8d38437b0cb with delete_type TABLET_DATA_DELETED (replaced by tablet 7431e9db4eca4dda856cece3204e666a) from {username='slave'} at 127.0.0.1:42036
I20240506 20:04:36.109366 24787 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c1b96ddace45fe8c59d8d38437b0cb with delete_type TABLET_DATA_DELETED (replaced by tablet 7431e9db4eca4dda856cece3204e666a) from {username='slave'} at 127.0.0.1:39578
I20240506 20:04:36.109802 25024 tablet_bootstrap.cc:654] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:36.109890 25026 tablet_bootstrap.cc:492] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9: Bootstrap starting.
W20240506 20:04:36.110312 24327 catalog_manager.cc:4810] TS b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): delete failed for tablet 04c1b96ddace45fe8c59d8d38437b0cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04c1b96ddace45fe8c59d8d38437b0cb already in progress: creating tablet
I20240506 20:04:36.110987 25030 ts_tablet_manager.cc:1875] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:36.111927 25028 tablet_bootstrap.cc:654] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:36.112751 25028 log.cc:824] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:36.112952 25030 ts_tablet_manager.cc:1888] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 20:04:36.113021 25030 log.cc:1196] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal/wals/04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:36.113233 25030 ts_tablet_manager.cc:1909] T 04c1b96ddace45fe8c59d8d38437b0cb P c35a1e4b657846ba945aa2602bea72cb: Deleting consensus metadata
I20240506 20:04:36.114404 24329 catalog_manager.cc:4830] TS c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): tablet 04c1b96ddace45fe8c59d8d38437b0cb (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
I20240506 20:04:36.114657 25026 tablet_bootstrap.cc:654] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9: Neither blocks nor log segments found. Creating new log.
W20240506 20:04:36.114881 24329 catalog_manager.cc:4797] TS c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): delete failed for tablet 04c1b96ddace45fe8c59d8d38437b0cb because the tablet was not found. No further retry: Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:36.115613 25026 log.cc:824] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9: Log is configured to *not* fsync() on all Append() calls
@ 0x7f5e24de2fc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240506 20:04:36.117040 25028 tablet_bootstrap.cc:492] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: No bootstrap required, opened a new log
I20240506 20:04:36.117198 25028 ts_tablet_manager.cc:1367] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
@ 0x555b1e6ee75c std::function<>::operator()() at ??:0
@ 0x7f5e212f4807 kudu::Thread::SuperviseThread() at ??:0
I20240506 20:04:36.119575 25028 raft_consensus.cc:348] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.119729 25028 raft_consensus.cc:374] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:36.119758 25028 raft_consensus.cc:729] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ecd08373ba4b2cb8623b3f4862bb98, State: Initialized, Role: FOLLOWER
I20240506 20:04:36.120146 25028 consensus_queue.cc:260] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.120998 25026 tablet_bootstrap.cc:492] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9: No bootstrap required, opened a new log
I20240506 20:04:36.121004 25028 ts_tablet_manager.cc:1398] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240506 20:04:36.121152 25026 ts_tablet_manager.cc:1367] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240506 20:04:36.121292 24853 heartbeater.cc:503] Master 127.23.94.124:36421 was elected leader, sending a full tablet report...
I20240506 20:04:36.123281 25028 tablet_bootstrap.cc:492] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: Bootstrap starting.
I20240506 20:04:36.124127 25026 raft_consensus.cc:348] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.124320 25026 raft_consensus.cc:374] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:36.124365 25026 raft_consensus.cc:729] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec4b56ff88545a69ab015a9d695d5d9, State: Initialized, Role: FOLLOWER
I20240506 20:04:36.124543 25028 tablet_bootstrap.cc:654] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: Neither blocks nor log segments found. Creating new log.
W20240506 20:04:36.125310 24329 catalog_manager.cc:4810] TS 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): delete failed for tablet 04c1b96ddace45fe8c59d8d38437b0cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04c1b96ddace45fe8c59d8d38437b0cb already in progress: creating tablet
I20240506 20:04:36.125887 24787 tablet_service.cc:1514] Processing DeleteTablet for tablet 04c1b96ddace45fe8c59d8d38437b0cb with delete_type TABLET_DATA_DELETED (replaced by tablet 7431e9db4eca4dda856cece3204e666a) from {username='slave'} at 127.0.0.1:39578
I20240506 20:04:36.126034 25024 tablet_bootstrap.cc:492] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: No bootstrap required, opened a new log
I20240506 20:04:36.126122 25024 ts_tablet_manager.cc:1367] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240506 20:04:36.126566 25024 raft_consensus.cc:348] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.126662 25024 raft_consensus.cc:374] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:36.126693 25024 raft_consensus.cc:729] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35a1e4b657846ba945aa2602bea72cb, State: Initialized, Role: FOLLOWER
@ 0x7f5e21b326db start_thread at ??:0
I20240506 20:04:36.126809 25024 consensus_queue.cc:260] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.126938 25028 tablet_bootstrap.cc:492] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: No bootstrap required, opened a new log
I20240506 20:04:36.127010 25028 ts_tablet_manager.cc:1367] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240506 20:04:36.127058 25024 ts_tablet_manager.cc:1398] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240506 20:04:36.127470 25028 raft_consensus.cc:348] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.127559 25028 raft_consensus.cc:374] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:36.127588 25028 raft_consensus.cc:729] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ecd08373ba4b2cb8623b3f4862bb98, State: Initialized, Role: FOLLOWER
I20240506 20:04:36.127694 25028 consensus_queue.cc:260] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.128012 25028 ts_tablet_manager.cc:1398] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240506 20:04:36.128233 25026 consensus_queue.cc:260] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
@ 0x7f5e1f06a71f clone at ??:0
I20240506 20:04:36.129438 25026 ts_tablet_manager.cc:1398] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240506 20:04:36.131681 25026 tablet_bootstrap.cc:492] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: Bootstrap starting.
I20240506 20:04:36.132082 24855 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:36.132561 25026 tablet_bootstrap.cc:654] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:36.132964 24951 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:53303; Asking this server to re-register.
I20240506 20:04:36.133620 25026 tablet_bootstrap.cc:492] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: No bootstrap required, opened a new log
I20240506 20:04:36.133651 24855 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:36.133687 25026 ts_tablet_manager.cc:1367] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 20:04:36.134238 25026 raft_consensus.cc:348] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.134325 25026 raft_consensus.cc:374] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:36.134353 25026 raft_consensus.cc:729] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec4b56ff88545a69ab015a9d695d5d9, State: Initialized, Role: FOLLOWER
I20240506 20:04:36.134433 25033 tablet_replica.cc:330] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: stopping tablet replica
I20240506 20:04:36.134486 25026 consensus_queue.cc:260] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.134683 25033 raft_consensus.cc:2229] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240506 20:04:36.134805 24951 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:36.134882 25026 ts_tablet_manager.cc:1398] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240506 20:04:36.134919 25033 raft_consensus.cc:2258] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:36.142045 25033 ts_tablet_manager.cc:1875] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:36.146292 25033 ts_tablet_manager.cc:1888] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 20:04:36.146375 25033 log.cc:1196] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal/wals/04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:36.146634 25033 ts_tablet_manager.cc:1909] T 04c1b96ddace45fe8c59d8d38437b0cb P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting consensus metadata
I20240506 20:04:36.188235 24198 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=906d0bea77ee4a8f9a2c24d07ff40655]
I20240506 20:04:36.188545 24198 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 4) took 1442 ms. Trace:
I20240506 20:04:36.188603 24198 rpcz_store.cc:276] 0506 20:04:34.746002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.747239 (+ 1237us) service_pool.cc:225] Handling call
0506 20:04:34.782136 (+ 34897us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.783395 (+ 1259us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.185599 (+402204us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.186079 (+ 480us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.186080 (+ 1us) write_op.cc:454] Released schema lock
0506 20:04:35.186196 (+ 116us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:36.188522 (+1002326us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.186042 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.186048 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.186051 (+ 3us) write_op.cc:454] Released schema lock
0506 20:04:35.186053 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1241129,"Hive Metastore.run_cpu_time_us":396,"Hive Metastore.run_wall_time_us":162875,"child_traces":[["op",{}]]}
W20240506 20:04:36.212417 24856 tablet.cc:2369] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 20:04:36.218086 25042 raft_consensus.cc:482] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:36.218309 25042 raft_consensus.cc:504] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.219602 25042 leader_election.cc:288] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:36.224695 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7431e9db4eca4dda856cece3204e666a" candidate_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" is_pre_election: true
I20240506 20:04:36.225000 24807 raft_consensus.cc:2454] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec4b56ff88545a69ab015a9d695d5d9 in term 0.
I20240506 20:04:36.224964 24669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7431e9db4eca4dda856cece3204e666a" candidate_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35a1e4b657846ba945aa2602bea72cb" is_pre_election: true
I20240506 20:04:36.225207 24669 raft_consensus.cc:2454] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec4b56ff88545a69ab015a9d695d5d9 in term 0.
I20240506 20:04:36.225669 24467 leader_election.cc:302] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9, b0ecd08373ba4b2cb8623b3f4862bb98; no voters:
I20240506 20:04:36.225924 25042 raft_consensus.cc:2781] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 20:04:36.226003 25042 raft_consensus.cc:482] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 20:04:36.226058 25042 raft_consensus.cc:3037] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:36.227171 25042 raft_consensus.cc:504] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.227519 25042 leader_election.cc:288] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 election: Requested vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:36.228506 24669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7431e9db4eca4dda856cece3204e666a" candidate_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c35a1e4b657846ba945aa2602bea72cb"
I20240506 20:04:36.228631 24669 raft_consensus.cc:3037] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:36.229760 24669 raft_consensus.cc:2454] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec4b56ff88545a69ab015a9d695d5d9 in term 1.
I20240506 20:04:36.230180 24469 leader_election.cc:302] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9, c35a1e4b657846ba945aa2602bea72cb; no voters:
I20240506 20:04:36.230464 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7431e9db4eca4dda856cece3204e666a" candidate_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98"
I20240506 20:04:36.230589 24807 raft_consensus.cc:3037] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:36.231655 24807 raft_consensus.cc:2454] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec4b56ff88545a69ab015a9d695d5d9 in term 1.
I20240506 20:04:36.232190 25042 raft_consensus.cc:2781] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 20:04:36.232290 25042 raft_consensus.cc:686] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 LEADER]: Becoming Leader. State: Replica: 3ec4b56ff88545a69ab015a9d695d5d9, State: Running, Role: LEADER
I20240506 20:04:36.232599 25042 consensus_queue.cc:237] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.241844 24201 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=83e3d80488894483b90b51b8631ea18e]
I20240506 20:04:36.242069 24201 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 2) took 1496 ms. Trace:
I20240506 20:04:36.242138 24201 rpcz_store.cc:276] 0506 20:04:34.745872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.746731 (+ 859us) service_pool.cc:225] Handling call
0506 20:04:34.782398 (+ 35667us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.783528 (+ 1130us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.205980 (+422452us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.206427 (+ 447us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.206428 (+ 1us) write_op.cc:454] Released schema lock
0506 20:04:35.206531 (+ 103us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:36.242055 (+1035524us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.206390 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.206396 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.206399 (+ 3us) write_op.cc:454] Released schema lock
0506 20:04:35.206402 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1383627,"Hive Metastore.run_cpu_time_us":411,"Hive Metastore.run_wall_time_us":73944,"spinlock_wait_cycles":66944,"child_traces":[["op",{}]]}
I20240506 20:04:36.289219 24193 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=e33e55f6620e4d90bfd0092c65c2aecd]
I20240506 20:04:36.289541 24193 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 9) took 1536 ms. Trace:
I20240506 20:04:36.289603 24193 rpcz_store.cc:276] 0506 20:04:34.752767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.753335 (+ 568us) service_pool.cc:225] Handling call
0506 20:04:34.782284 (+ 28949us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.783594 (+ 1310us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.227316 (+443722us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.227735 (+ 419us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.227736 (+ 1us) write_op.cc:454] Released schema lock
0506 20:04:35.227835 (+ 99us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:36.289519 (+1061684us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.227698 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.227704 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.227708 (+ 4us) write_op.cc:454] Released schema lock
0506 20:04:35.227710 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1436270,"Hive Metastore.run_cpu_time_us":387,"Hive Metastore.run_wall_time_us":68607,"child_traces":[["op",{}]]}
W20240506 20:04:36.291733 24718 tablet.cc:2369] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 20:04:36.342267 24194 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=4b49fca39a0a4f789d4cd4f791063786]
I20240506 20:04:36.342567 24194 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 8) took 1589 ms. Trace:
I20240506 20:04:36.342651 24194 rpcz_store.cc:276] 0506 20:04:34.752715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.753084 (+ 369us) service_pool.cc:225] Handling call
0506 20:04:34.782533 (+ 29449us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.783795 (+ 1262us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.262028 (+478233us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.262522 (+ 494us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.262523 (+ 1us) write_op.cc:454] Released schema lock
0506 20:04:35.262632 (+ 109us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:36.342540 (+1079908us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.262481 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.262488 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.262491 (+ 3us) write_op.cc:454] Released schema lock
0506 20:04:35.262494 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1469953,"Hive Metastore.run_cpu_time_us":389,"Hive Metastore.run_wall_time_us":87683,"child_traces":[["op",{}]]}
W20240506 20:04:36.370954 24583 tablet.cc:2369] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 20:04:36.387832 24200 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=eb6dd8053f18430da37b1eb0a3931948]
I20240506 20:04:36.388124 24200 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 1) took 1642 ms. Trace:
I20240506 20:04:36.388196 24200 rpcz_store.cc:276] 0506 20:04:34.745794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.746095 (+ 301us) service_pool.cc:225] Handling call
0506 20:04:34.783678 (+ 37583us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.783933 (+ 255us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.284896 (+500963us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.285460 (+ 564us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.285462 (+ 2us) write_op.cc:454] Released schema lock
0506 20:04:35.285613 (+ 151us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:36.388103 (+1102490us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.285402 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.285412 (+ 10us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.285416 (+ 4us) write_op.cc:454] Released schema lock
0506 20:04:35.285420 (+ 4us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1534608,"Hive Metastore.run_cpu_time_us":395,"Hive Metastore.run_wall_time_us":68342,"spinlock_wait_cycles":6784,"child_traces":[["op",{}]]}
W20240506 20:04:36.388391 24861 connection.cc:535] client connection to 127.23.94.124:36421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:36.388725 24861 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.23.94.124:36421)
W20240506 20:04:36.389214 24861 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.23.94.124:36421)
W20240506 20:04:36.389583 24603 connection.cc:535] client connection to 127.23.94.124:36421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:36.389586 24741 connection.cc:535] client connection to 127.23.94.124:36421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:36.389925 24468 connection.cc:535] client connection to 127.23.94.124:36421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:36.392647 24741 connection.cc:535] server connection from 127.0.0.1:39578 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:36.393772 24913 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 20:04:36.390284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.94.124:36421 (local address 127.23.94.67:54517)
0506 20:04:36.390461 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0506 20:04:36.390476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0506 20:04:36.390571 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 20:04:36.393214 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":98}
W20240506 20:04:36.394577 25053 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 20:04:36.390375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.94.124:36421 (local address 127.23.94.65:39533)
0506 20:04:36.390540 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0506 20:04:36.390551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0506 20:04:36.390613 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 20:04:36.394153 (+ 3540us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":81}
W20240506 20:04:36.395617 24988 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 20:04:36.391042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.94.124:36421 (local address 127.0.0.1:60432)
0506 20:04:36.391193 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0506 20:04:36.391206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0506 20:04:36.391279 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 20:04:36.394983 (+ 3704us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":53}
I20240506 20:04:36.422606 24196 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=fced752d1b68486d8b91c515f1181350]
I20240506 20:04:36.422861 24196 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 6) took 1670 ms. Trace:
I20240506 20:04:36.422950 24196 rpcz_store.cc:276] 0506 20:04:34.752537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.752601 (+ 64us) service_pool.cc:225] Handling call
0506 20:04:34.781899 (+ 29298us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.782874 (+ 975us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.306857 (+523983us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.307368 (+ 511us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.307369 (+ 1us) write_op.cc:454] Released schema lock
0506 20:04:35.307476 (+ 107us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:36.422846 (+1115370us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.307328 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.307334 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.307338 (+ 4us) write_op.cc:454] Released schema lock
0506 20:04:35.307340 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1582062,"Hive Metastore.run_cpu_time_us":429,"Hive Metastore.run_wall_time_us":56635,"mutex_wait_us":902,"spinlock_wait_cycles":22400,"child_traces":[["op",{}]]}
I20240506 20:04:36.432376 25055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:36.432603 25055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:36.434666 24210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" is_pre_election: true
I20240506 20:04:36.435650 25055 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:36.435789 24929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
W20240506 20:04:36.437762 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:36.437852 24929 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:36.438120 25055 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 20:04:36.454785 24192 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=4935d9be7af749d8863a707db7336e33]
I20240506 20:04:36.455040 24192 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 10) took 1700 ms. Trace:
I20240506 20:04:36.455111 24192 rpcz_store.cc:276] 0506 20:04:34.754255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.754312 (+ 57us) service_pool.cc:225] Handling call
0506 20:04:34.783893 (+ 29581us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.784147 (+ 254us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.333255 (+549108us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.333790 (+ 535us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.333792 (+ 2us) write_op.cc:454] Released schema lock
0506 20:04:35.333956 (+ 164us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:36.455025 (+1121069us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.333737 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.333746 (+ 9us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.333750 (+ 4us) write_op.cc:454] Released schema lock
0506 20:04:35.333753 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1611197,"Hive Metastore.run_cpu_time_us":395,"Hive Metastore.run_wall_time_us":58475,"spinlock_wait_cycles":70528,"child_traces":[["op",{}]]}
I20240506 20:04:36.489364 24199 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=59d2865ad3bc4f8da8ba8011111472c7]
I20240506 20:04:36.489591 24199 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53480 (request call id 3) took 1743 ms. Trace:
I20240506 20:04:36.489639 24199 rpcz_store.cc:276] 0506 20:04:34.745957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 20:04:34.746997 (+ 1040us) service_pool.cc:225] Handling call
0506 20:04:34.784042 (+ 37045us) catalog_manager.cc:2072] Acquired catalog manager lock
0506 20:04:34.784296 (+ 254us) catalog_manager.cc:2118] Created new table and tablet info
0506 20:04:35.352673 (+568377us) catalog_manager.cc:2144] Created new table in HMS catalog
0506 20:04:35.353201 (+ 528us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.353203 (+ 2us) write_op.cc:454] Released schema lock
0506 20:04:35.353318 (+ 115us) catalog_manager.cc:2152] Rolling back HMS table creation
0506 20:04:36.489581 (+1136263us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 20:04:35.353153 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 25192b33b13d4b089dc3fb1d1f6fc2e9 is not leader of this config: current role FOLLOWER)
0506 20:04:35.353161 (+ 8us) write_op.cc:489] Releasing partition, row and schema locks
0506 20:04:35.353165 (+ 4us) write_op.cc:454] Released schema lock
0506 20:04:35.353168 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1650249,"Hive Metastore.run_cpu_time_us":394,"Hive Metastore.run_wall_time_us":53899,"spinlock_wait_cycles":15488,"child_traces":[["op",{}]]}
I20240506 20:04:36.522974 25042 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:36.523090 25042 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:36.523452 25042 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:04:36.524153 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:04:36.524292 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:36.524379 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:04:36.524528 25042 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:04:36.605930 25057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 038cb801e9214358bab57093f322c097)
I20240506 20:04:36.606036 25057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:36.606365 25057 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:36.607426 24186 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:36.610137 24961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029" is_pre_election: true
I20240506 20:04:36.610357 24961 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 2.
I20240506 20:04:36.610626 24185 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097
I20240506 20:04:36.610766 25057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 20:04:36.610827 25057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 038cb801e9214358bab57093f322c097)
I20240506 20:04:36.610853 25057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 3 FOLLOWER]: Advancing to term 4
I20240506 20:04:36.611598 25057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:36.611882 25057 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 4 election: Requested vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:36.612123 24961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029"
I20240506 20:04:36.612241 24961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 2 FOLLOWER]: Advancing to term 4
W20240506 20:04:36.612679 24186 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:36.614024 24961 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 4.
I20240506 20:04:36.614269 24185 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097
I20240506 20:04:36.614414 25057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 4 FOLLOWER]: Leader election won for term 4
I20240506 20:04:36.614542 25057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 4 LEADER]: Becoming Leader. State: Replica: 25192b33b13d4b089dc3fb1d1f6fc2e9, State: Running, Role: LEADER
I20240506 20:04:36.614672 25057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:36.615177 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25192b33b13d4b089dc3fb1d1f6fc2e9. Latest consensus state: current_term: 4 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:36.615262 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:36.615478 25062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:04:36.616569 25062 catalog_manager.cc:669] Loaded metadata for table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]
I20240506 20:04:36.617484 25062 tablet_loader.cc:96] loaded metadata for tablet 7431e9db4eca4dda856cece3204e666a (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da])
I20240506 20:04:36.617583 25062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:04:36.617967 25062 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:36.618011 25062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:04:36.618605 25062 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:04:36.618956 25062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Loaded TSK: 0
I20240506 20:04:36.619364 25062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:04:36.619693 25062 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:04:36.620019 25062 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 11
I20240506 20:04:36.629884 24199 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_e80158bfa50b427cbd790f160164b525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:36.629941 24192 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_7f7588f0e29e4057a66f08687ec28975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:36.630035 24194 master_service.cc:946] ReplaceTablet: received request to replace tablet 7431e9db4eca4dda856cece3204e666a from {username='slave'} at 127.0.0.1:53480
I20240506 20:04:36.630203 24200 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_6d90713e02f146279b23811c97941079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:36.630363 24193 master_service.cc:946] ReplaceTablet: received request to replace tablet 7431e9db4eca4dda856cece3204e666a from {username='slave'} at 127.0.0.1:53480
I20240506 20:04:36.630487 24198 master_service.cc:946] ReplaceTablet: received request to replace tablet 7431e9db4eca4dda856cece3204e666a from {username='slave'} at 127.0.0.1:53480
I20240506 20:04:36.630499 24196 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_8df5615aacee4940a5402206a921f0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:36.630520 24195 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_6e4d3650620d474ab3107f0f53f5e807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:36.630707 24201 master_service.cc:946] ReplaceTablet: received request to replace tablet 7431e9db4eca4dda856cece3204e666a from {username='slave'} at 127.0.0.1:53480
I20240506 20:04:36.630532 24197 master_service.cc:946] ReplaceTablet: received request to replace tablet 7431e9db4eca4dda856cece3204e666a from {username='slave'} at 127.0.0.1:53480
I20240506 20:04:36.632885 24961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Refusing update from remote peer 25192b33b13d4b089dc3fb1d1f6fc2e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 12. (index mismatch)
I20240506 20:04:36.633397 25061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 4, Time since last communication: 0.000s
W20240506 20:04:36.633612 24186 consensus_peers.cc:485] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 -> Peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Couldn't send request to peer 038cb801e9214358bab57093f322c097. Status: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 20:04:36.636478 24401 catalog_manager.cc:2706] Deleting table default.table_c568fe36c59c4d93bda72f412845b2f2 [id=f8c2559276c74cc698dc2e82a2d7bf5f] in response to Hive Metastore notification log event 12
W20240506 20:04:36.636698 24401 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_c568fe36c59c4d93bda72f412845b2f2: Not found: the table does not exist: table_id: "f8c2559276c74cc698dc2e82a2d7bf5f" table_name: "default.table_c568fe36c59c4d93bda72f412845b2f2"
F20240506 20:04:36.636758 24401 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 12): 12 CREATE_TABLE default.table_c66444cb675241d1a23c4348383b208e
*** Check failure stack trace: ***
I20240506 20:04:36.637672 25066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:36.637773 25066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:36.638022 25055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25192b33b13d4b089dc3fb1d1f6fc2e9. Latest consensus state: current_term: 4 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:36.638116 25055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:36.639477 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:36.639575 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:36.639864 25055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:36.639964 25055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:36.640507 24194 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Sending DeleteTablet for 3 replicas of tablet 7431e9db4eca4dda856cece3204e666a
I20240506 20:04:36.641593 25064 mvcc.cc:205] Tried to move back new op lower bound from 7024745990682255360 to 7024745985318817792. Current Snapshot: MvccSnapshot[applied={T|T < 7024745990682255360}]
I20240506 20:04:36.641623 24194 catalog_manager.cc:6554] ReplaceTablet: tablet 7431e9db4eca4dda856cece3204e666a deleted and replaced by tablet af340b9943944cbb84a445e6c0034bc5
I20240506 20:04:36.641856 24194 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53480:
table { table_name: "default.Table_c66444cb675241d1a23c4348383b208e" } new_table_name: "default.Table_ea33dccc2852419e87b57fc573f6572e" modify_external_catalogs: true
I20240506 20:04:36.642275 24198 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_2ee32586537f466fa07ee5ad6c48dec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:36.642659 24201 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_638fa9e2f7b349a38f86c3d585998fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:36.643038 24197 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_758d3e4f504a489eb12b7bf93d583cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:36.641558 24193 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53480:
name: "default.Table_c568fe36c59c4d93bda72f412845b2f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f60541d2dcd google::LogMessage::Fail() at ??:0
I20240506 20:04:36.645910 24514 tablet_service.cc:1514] Processing DeleteTablet for tablet 7431e9db4eca4dda856cece3204e666a with delete_type TABLET_DATA_DELETED (replaced by tablet af340b9943944cbb84a445e6c0034bc5) from {username='slave'} at 127.0.0.1:54348
I20240506 20:04:36.648406 24514 tablet_service.cc:1467] Processing CreateTablet for tablet af340b9943944cbb84a445e6c0034bc5 (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f60541d6b93 google::LogMessage::SendToLog() at ??:0
I20240506 20:04:36.648699 24514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af340b9943944cbb84a445e6c0034bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:36.648872 24649 tablet_service.cc:1514] Processing DeleteTablet for tablet 7431e9db4eca4dda856cece3204e666a with delete_type TABLET_DATA_DELETED (replaced by tablet af340b9943944cbb84a445e6c0034bc5) from {username='slave'} at 127.0.0.1:42046
I20240506 20:04:36.649230 24648 tablet_service.cc:1467] Processing CreateTablet for tablet af340b9943944cbb84a445e6c0034bc5 (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:36.649456 24648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af340b9943944cbb84a445e6c0034bc5. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f60541d27cc google::LogMessage::Flush() at ??:0
I20240506 20:04:36.651733 24787 tablet_service.cc:1514] Processing DeleteTablet for tablet 7431e9db4eca4dda856cece3204e666a with delete_type TABLET_DATA_DELETED (replaced by tablet af340b9943944cbb84a445e6c0034bc5) from {username='slave'} at 127.0.0.1:39592
I20240506 20:04:36.651805 25076 tablet_bootstrap.cc:492] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: Bootstrap starting.
@ 0x7f60541d2a11 google::LogMessage::~LogMessage() at ??:0
I20240506 20:04:36.651974 25078 tablet_replica.cc:330] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: stopping tablet replica
I20240506 20:04:36.651950 24786 tablet_service.cc:1467] Processing CreateTablet for tablet af340b9943944cbb84a445e6c0034bc5 (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:36.652068 25078 raft_consensus.cc:2229] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 20:04:36.652160 25078 raft_consensus.cc:2258] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:36.652171 24786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af340b9943944cbb84a445e6c0034bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:36.652511 25078 ts_tablet_manager.cc:1875] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:36.652837 25076 tablet_bootstrap.cc:654] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:36.653510 25077 tablet_bootstrap.cc:492] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: Bootstrap starting.
I20240506 20:04:36.654434 25077 tablet_bootstrap.cc:654] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:36.654466 25078 ts_tablet_manager.cc:1888] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 20:04:36.654512 25078 log.cc:1196] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal/wals/7431e9db4eca4dda856cece3204e666a
I20240506 20:04:36.654659 25078 ts_tablet_manager.cc:1909] T 7431e9db4eca4dda856cece3204e666a P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting consensus metadata
I20240506 20:04:36.655107 25075 tablet_replica.cc:330] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: stopping tablet replica
I20240506 20:04:36.655216 25075 raft_consensus.cc:2229] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 20:04:36.655309 25075 raft_consensus.cc:2258] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:36.655349 24185 catalog_manager.cc:4830] TS b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): tablet 7431e9db4eca4dda856cece3204e666a (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
@ 0x7f60590fe4f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240506 20:04:36.655616 25075 ts_tablet_manager.cc:1875] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:36.656961 25073 tablet_replica.cc:330] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: stopping tablet replica
I20240506 20:04:36.657182 25073 raft_consensus.cc:2229] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 LEADER]: Raft consensus shutting down.
I20240506 20:04:36.657421 25073 pending_rounds.cc:62] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: Trying to abort 1 pending ops.
I20240506 20:04:36.657465 25073 pending_rounds.cc:68] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024745989052022784 op_type: NO_OP noop_request { }
I20240506 20:04:36.657529 25073 raft_consensus.cc:2866] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 20:04:36.657568 25073 raft_consensus.cc:2258] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:36.657547 25075 ts_tablet_manager.cc:1888] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240506 20:04:36.657605 25075 log.cc:1196] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal/wals/7431e9db4eca4dda856cece3204e666a
I20240506 20:04:36.657795 25075 ts_tablet_manager.cc:1909] T 7431e9db4eca4dda856cece3204e666a P c35a1e4b657846ba945aa2602bea72cb: Deleting consensus metadata
I20240506 20:04:36.658252 25076 tablet_bootstrap.cc:492] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: No bootstrap required, opened a new log
I20240506 20:04:36.658331 25076 ts_tablet_manager.cc:1367] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240506 20:04:36.658489 25073 ts_tablet_manager.cc:1875] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:36.658789 25076 raft_consensus.cc:348] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:36.658890 25076 raft_consensus.cc:374] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:36.658918 25076 raft_consensus.cc:729] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec4b56ff88545a69ab015a9d695d5d9, State: Initialized, Role: FOLLOWER
@ 0x7f60590fd5ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240506 20:04:36.659044 25076 consensus_queue.cc:260] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:36.659286 25076 ts_tablet_manager.cc:1398] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240506 20:04:36.660197 25073 ts_tablet_manager.cc:1888] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 20:04:36.660270 25073 log.cc:1196] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal/wals/7431e9db4eca4dda856cece3204e666a
I20240506 20:04:36.660466 25077 tablet_bootstrap.cc:492] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: No bootstrap required, opened a new log
I20240506 20:04:36.660542 25077 ts_tablet_manager.cc:1367] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240506 20:04:36.660542 25073 ts_tablet_manager.cc:1909] T 7431e9db4eca4dda856cece3204e666a P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting consensus metadata
I20240506 20:04:36.661085 25077 raft_consensus.cc:348] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:36.661180 25077 raft_consensus.cc:374] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:36.661211 25077 raft_consensus.cc:729] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35a1e4b657846ba945aa2602bea72cb, State: Initialized, Role: FOLLOWER
I20240506 20:04:36.661324 25077 consensus_queue.cc:260] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:36.661466 24187 catalog_manager.cc:4830] TS 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): tablet 7431e9db4eca4dda856cece3204e666a (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
I20240506 20:04:36.661660 24716 heartbeater.cc:503] Master 127.23.94.126:40141 was elected leader, sending a full tablet report...
I20240506 20:04:36.661813 25077 ts_tablet_manager.cc:1398] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240506 20:04:36.663056 24187 catalog_manager.cc:4830] TS c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): tablet 7431e9db4eca4dda856cece3204e666a (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
I20240506 20:04:36.663105 25080 tablet_bootstrap.cc:492] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: Bootstrap starting.
I20240506 20:04:36.664018 25080 tablet_bootstrap.cc:654] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:36.665022 25080 tablet_bootstrap.cc:492] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: No bootstrap required, opened a new log
I20240506 20:04:36.665094 25080 ts_tablet_manager.cc:1367] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
@ 0x7f60590fcd51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20240506 20:04:36.665506 25080 raft_consensus.cc:348] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:36.665594 25080 raft_consensus.cc:374] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:36.665621 25080 raft_consensus.cc:729] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ecd08373ba4b2cb8623b3f4862bb98, State: Initialized, Role: FOLLOWER
I20240506 20:04:36.665750 25080 consensus_queue.cc:260] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:36.666047 25080 ts_tablet_manager.cc:1398] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240506 20:04:36.666206 24854 heartbeater.cc:503] Master 127.23.94.126:40141 was elected leader, sending a full tablet report...
@ 0x7f6059101fc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56249f52b75c std::function<>::operator()() at ??:0
@ 0x7f6055613807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6055e516db start_thread at ??:0
@ 0x7f605338971f clone at ??:0
I20240506 20:04:36.675398 25082 raft_consensus.cc:482] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:36.675483 25082 raft_consensus.cc:504] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:36.676455 25082 leader_election.cc:288] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:36.680606 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af340b9943944cbb84a445e6c0034bc5" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" is_pre_election: true
I20240506 20:04:36.680789 24807 raft_consensus.cc:2454] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 0.
I20240506 20:04:36.681269 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af340b9943944cbb84a445e6c0034bc5" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" is_pre_election: true
I20240506 20:04:36.681327 24602 leader_election.cc:302] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb; no voters:
I20240506 20:04:36.681447 24534 raft_consensus.cc:2454] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 0.
I20240506 20:04:36.681571 25082 raft_consensus.cc:2781] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 20:04:36.681622 25082 raft_consensus.cc:482] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 20:04:36.681655 25082 raft_consensus.cc:3037] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:36.682585 25082 raft_consensus.cc:504] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:36.682907 25082 leader_election.cc:288] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 election: Requested vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:36.683270 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af340b9943944cbb84a445e6c0034bc5" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98"
I20240506 20:04:36.683370 24807 raft_consensus.cc:3037] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:36.683374 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af340b9943944cbb84a445e6c0034bc5" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec4b56ff88545a69ab015a9d695d5d9"
I20240506 20:04:36.683475 24534 raft_consensus.cc:3037] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:36.684242 24807 raft_consensus.cc:2454] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 1.
I20240506 20:04:36.684358 24534 raft_consensus.cc:2454] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 1.
I20240506 20:04:36.684568 24602 leader_election.cc:302] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb; no voters:
I20240506 20:04:36.684765 25082 raft_consensus.cc:2781] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Leader election won for term 1
I20240506 20:04:36.684931 25082 raft_consensus.cc:686] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 1 LEADER]: Becoming Leader. State: Replica: c35a1e4b657846ba945aa2602bea72cb, State: Running, Role: LEADER
I20240506 20:04:36.685178 25082 consensus_queue.cc:237] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
W20240506 20:04:36.817198 24988 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 20:04:36.816600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.94.126:40141 (local address 127.0.0.1:53492)
0506 20:04:36.817059 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0506 20:04:36.817106 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":367}
W20240506 20:04:36.817451 25089 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 20:04:36.815939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.94.126:40141 (local address 127.23.94.66:58093)
0506 20:04:36.816137 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0506 20:04:36.816152 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0506 20:04:36.816245 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 20:04:36.816843 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":111}
I20240506 20:04:37.108044 25082 consensus_queue.cc:1035] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 20:04:37.112327 25103 consensus_queue.cc:1035] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 20:04:37.147914 25109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:37.148027 25109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:37.148432 25109 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:37.149474 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:37.149564 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:37.149603 24929 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:37.149741 25109 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 20:04:37.682608 25120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:37.682730 25120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:37.683117 25120 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:37.684213 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:37.684316 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:37.684365 24929 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:37.684504 25120 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 20:04:38.336345 25126 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:38.336468 25126 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:38.336891 25126 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:04:38.337529 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:04:38.337702 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:38.337790 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:04:38.337903 25126 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:04:38.516461 25127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:38.516580 25127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:38.517006 25127 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:38.518270 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:38.518391 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:38.518445 24929 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:38.518568 25127 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 20:04:38.997087 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--webserver_interface=127.23.94.124
--webserver_port=36977
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:04:39.097352 25135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:39.097549 25135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:39.097591 25135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:39.100533 25135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:39.100582 25135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:39.100602 25135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:39.100656 25135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:39.104182 25135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.94.124
--webserver_port=36977
--never_fsync=true
--heap_profile_path=/tmp/kudu.25135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:39.104965 25135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:39.105960 25135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:39.112648 25140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:39.113013 25141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:39.113330 25143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:39.113790 25135 server_base.cc:1034] running on GCE node
I20240506 20:04:39.114212 25135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:39.114971 25135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:39.116118 25135 hybrid_clock.cc:648] HybridClock initialized: now 1715025879116106 us; error 30 us; skew 500 ppm
I20240506 20:04:39.123692 25135 webserver.cc:460] Webserver started at http://127.23.94.124:36977/ using document root <none> and password file <none>
I20240506 20:04:39.124204 25135 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:39.124269 25135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:39.127336 25135 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20240506 20:04:39.129094 25151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:39.130509 25135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:39.130610 25135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
uuid: "038cb801e9214358bab57093f322c097"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:39.130962 25135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:39.145366 25156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:39.145462 25156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:39.145808 25156 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:39.146785 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:39.146891 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:39.146934 24929 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:39.147058 25156 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 20:04:39.149612 25135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:39.150169 25135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:39.150301 25135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:39.156436 25207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.124:36421 every 8 connection(s)
I20240506 20:04:39.156448 25135 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.124:36421
I20240506 20:04:39.157444 25135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
I20240506 20:04:39.161988 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 25135
I20240506 20:04:39.162235 25208 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:04:39.164637 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap starting.
I20240506 20:04:39.169553 24715 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:39.170985 25208 log.cc:824] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:39.178196 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=7 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 20:04:39.178465 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap complete.
I20240506 20:04:39.180821 25208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:39.181124 25208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038cb801e9214358bab57093f322c097, State: Initialized, Role: FOLLOWER
I20240506 20:04:39.181495 25208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:39.182116 25215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:39.182216 25215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:39.182374 25208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:04:39.188061 25229 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:39.188177 25229 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: loading cluster ID for follower catalog manager: success
I20240506 20:04:39.186396 25135 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:04:39.188455 25135 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:39.190100 25229 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: acquiring CA information for follower catalog manager: success
I20240506 20:04:39.191417 25229 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:04:39.593729 25215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:39.593916 25215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:39.595028 25215 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:39.595507 25153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:39.596314 25153 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:39.598526 24961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029" is_pre_election: true
I20240506 20:04:39.598686 24961 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 038cb801e9214358bab57093f322c097 in current term 4: Already voted for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in this term.
I20240506 20:04:39.599107 25152 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:39.599329 25215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 3 FOLLOWER]: Advancing to term 4
I20240506 20:04:39.601015 25215 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 20:04:39.623487 24580 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:39.623963 24853 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:39.624821 25171 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:32803; Asking this server to re-register.
I20240506 20:04:39.624825 25172 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:55259; Asking this server to re-register.
I20240506 20:04:39.625231 24853 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:39.625257 24580 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:39.626323 25172 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:39.626423 25171 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:39.662820 24715 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:39.663602 25172 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:38971; Asking this server to re-register.
I20240506 20:04:39.663926 24715 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:39.664575 25172 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:40.269763 25238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:40.269872 25238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:40.270275 25238 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
I20240506 20:04:40.270653 24961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 5 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029" is_pre_election: true
I20240506 20:04:40.270833 24961 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 038cb801e9214358bab57093f322c097 for term 5 because replica has last-logged OpId of term: 4 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 10.
W20240506 20:04:40.271342 25153 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:40.271421 25153 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:40.271556 25238 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 20:04:41.231194 25247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:41.231334 25247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:41.231761 25247 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:41.233100 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:41.235487 25182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 5 candidate_status { last_received { term: 4 index: 13 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097" is_pre_election: true
I20240506 20:04:41.235716 25182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853e921441fd4ee4b5bb84fdab584029 in term 4.
I20240506 20:04:41.235988 24929 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:41.236153 25247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 20:04:41.236210 25247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:04:41.236243 25247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 4 FOLLOWER]: Advancing to term 5
I20240506 20:04:41.237043 25247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:41.237334 25247 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 election: Requested vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:41.237753 25182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 5 candidate_status { last_received { term: 4 index: 13 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097"
I20240506 20:04:41.237855 25182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 4 FOLLOWER]: Advancing to term 5
W20240506 20:04:41.238092 24929 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:41.238672 25182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853e921441fd4ee4b5bb84fdab584029 in term 5.
I20240506 20:04:41.238965 24929 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:41.239101 25247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 20:04:41.239284 25247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 LEADER]: Becoming Leader. State: Replica: 853e921441fd4ee4b5bb84fdab584029, State: Running, Role: LEADER
I20240506 20:04:41.239544 25247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.13, Last appended by leader: 13, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:41.240484 25252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 853e921441fd4ee4b5bb84fdab584029. Latest consensus state: current_term: 5 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:41.240588 25252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:41.240873 25254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:04:41.242767 25254 catalog_manager.cc:669] Loaded metadata for table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]
I20240506 20:04:41.243889 25254 tablet_loader.cc:96] loaded metadata for tablet af340b9943944cbb84a445e6c0034bc5 (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da])
I20240506 20:04:41.244010 25254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:04:41.244400 25254 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:41.244441 25254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:04:41.244963 25254 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:04:41.245354 25254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Loaded TSK: 0
I20240506 20:04:41.245967 25254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:04:41.246326 25254 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:04:41.246672 25254 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 11
I20240506 20:04:41.339071 24948 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_2ee32586537f466fa07ee5ad6c48dec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:41.339352 24944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_6e4d3650620d474ab3107f0f53f5e807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:41.339221 24946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_c568fe36c59c4d93bda72f412845b2f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:41.338799 24951 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_a1b4d3a0d5ce479584c8f895844cadc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:41.339175 24942 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_7f7588f0e29e4057a66f08687ec28975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:41.339340 24945 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_8df5615aacee4940a5402206a921f0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:41.338982 24947 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57960:
table { table_name: "default.Table_c66444cb675241d1a23c4348383b208e" } new_table_name: "default.Table_ea33dccc2852419e87b57fc573f6572e" modify_external_catalogs: true
I20240506 20:04:41.339432 24943 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_6d90713e02f146279b23811c97941079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:41.338953 24949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_638fa9e2f7b349a38f86c3d585998fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:41.338799 24950 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_758d3e4f504a489eb12b7bf93d583cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:41.347091 25010 catalog_manager.cc:2706] Deleting table default.table_c568fe36c59c4d93bda72f412845b2f2 [id=f8c2559276c74cc698dc2e82a2d7bf5f] in response to Hive Metastore notification log event 12
W20240506 20:04:41.347347 25010 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_c568fe36c59c4d93bda72f412845b2f2: Not found: the table does not exist: table_id: "f8c2559276c74cc698dc2e82a2d7bf5f" table_name: "default.table_c568fe36c59c4d93bda72f412845b2f2"
F20240506 20:04:41.347404 25010 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 12): 12 CREATE_TABLE default.table_c66444cb675241d1a23c4348383b208e
*** Check failure stack trace: ***
@ 0x7f022285fdcd google::LogMessage::Fail() at ??:0
@ 0x7f0222863b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f022285f7cc google::LogMessage::Flush() at ??:0
@ 0x7f022285fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f022778b4f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f022778a5ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f0227789d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7f022778efc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55bd1b1b775c std::function<>::operator()() at ??:0
@ 0x7f0223ca0807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f02244de6db start_thread at ??:0
@ 0x7f0221a1671f clone at ??:0
W20240506 20:04:41.513000 24860 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.23.94.125:42097)
W20240506 20:04:41.513372 24860 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.94.125:42097)
W20240506 20:04:41.513847 24860 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.23.94.125:42097)
I20240506 20:04:41.551443 25268 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:41.551556 25268 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:41.551913 25268 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:04:41.552513 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:04:41.552608 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:41.552721 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:04:41.552867 25268 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:04:41.667414 23929 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3taq95/build/debug/bin/kudu with pid 25135
W20240506 20:04:41.672245 24861 connection.cc:535] client connection to 127.23.94.124:36421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:41.672497 24603 connection.cc:535] client connection to 127.23.94.124:36421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 20:04:42.230238 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--webserver_interface=127.23.94.124
--webserver_port=36977
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:04:42.332890 25273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:42.333091 25273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:42.333155 25273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:42.336094 25273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:42.336146 25273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:42.336170 25273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:42.336192 25273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:42.339637 25273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.94.124
--webserver_port=36977
--never_fsync=true
--heap_profile_path=/tmp/kudu.25273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:42.340389 25273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:42.341428 25273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:42.348064 25278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:42.348341 25279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:42.348691 25281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:42.349097 25273 server_base.cc:1034] running on GCE node
I20240506 20:04:42.349483 25273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:42.350286 25273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:42.351431 25273 hybrid_clock.cc:648] HybridClock initialized: now 1715025882351416 us; error 28 us; skew 500 ppm
I20240506 20:04:42.353088 25273 webserver.cc:460] Webserver started at http://127.23.94.124:36977/ using document root <none> and password file <none>
I20240506 20:04:42.353567 25273 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:42.353621 25273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:42.356510 25273 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240506 20:04:42.358214 25287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:42.359058 25273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 20:04:42.359155 25273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
uuid: "038cb801e9214358bab57093f322c097"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:42.359503 25273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:42.378087 25273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:42.378631 25273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:42.378772 25273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:42.384594 25273 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.124:36421
I20240506 20:04:42.384599 25339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.124:36421 every 8 connection(s)
I20240506 20:04:42.385622 25273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
I20240506 20:04:42.389348 25340 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:04:42.390834 25340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap starting.
I20240506 20:04:42.395135 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 25273
I20240506 20:04:42.396045 25340 log.cc:824] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:42.404645 25340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=7 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 20:04:42.404992 25340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap complete.
I20240506 20:04:42.408072 25340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:42.408442 25340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038cb801e9214358bab57093f322c097, State: Initialized, Role: FOLLOWER
I20240506 20:04:42.408943 25340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:42.409318 25344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:42.409425 25344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:42.409871 25340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:04:42.414968 25358 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:42.415055 25358 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: loading cluster ID for follower catalog manager: success
I20240506 20:04:42.413993 25273 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:04:42.416002 25273 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:42.417182 25358 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: acquiring CA information for follower catalog manager: success
I20240506 20:04:42.418269 25358 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:04:42.635572 24580 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:42.635656 24853 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:42.636958 25304 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:33079; Asking this server to re-register.
I20240506 20:04:42.636968 25303 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:38801; Asking this server to re-register.
I20240506 20:04:42.637475 24853 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:42.638121 24580 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:42.638495 25304 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:42.638818 25303 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:42.676332 24715 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:42.677045 25303 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:33619; Asking this server to re-register.
I20240506 20:04:42.677382 24715 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:42.678081 25303 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:42.848086 25344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:42.848277 25344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:42.849395 25344 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:42.849789 25288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:42.850564 25288 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:42.850662 25289 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:42.850766 25289 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:42.850970 25344 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 20:04:43.384581 25366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:43.384743 25366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:43.385190 25366 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:43.386361 25288 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:43.386512 25289 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:43.386595 25289 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:43.386727 25366 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 20:04:44.264218 25374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:44.264338 25374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:44.264778 25374 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:44.265893 25289 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:44.265974 25288 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:44.266059 25288 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:44.266197 25374 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 20:04:44.456707 23929 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3taq95/build/debug/bin/kudu with pid 25273
I20240506 20:04:45.201879 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--webserver_interface=127.23.94.124
--webserver_port=36977
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:04:45.300184 25382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:45.300364 25382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:45.300418 25382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:45.303371 25382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:45.303419 25382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:45.303444 25382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:45.303467 25382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:45.306928 25382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.94.124
--webserver_port=36977
--never_fsync=true
--heap_profile_path=/tmp/kudu.25382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:45.307634 25382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:45.308566 25382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:45.314998 25388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:45.315052 25387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:45.315448 25390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:45.315831 25382 server_base.cc:1034] running on GCE node
I20240506 20:04:45.316247 25382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:45.317060 25382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:45.318192 25382 hybrid_clock.cc:648] HybridClock initialized: now 1715025885318183 us; error 23 us; skew 500 ppm
I20240506 20:04:45.324684 25382 webserver.cc:460] Webserver started at http://127.23.94.124:36977/ using document root <none> and password file <none>
I20240506 20:04:45.325148 25382 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:45.325199 25382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:45.328001 25382 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240506 20:04:45.329674 25396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:45.330796 25382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 20:04:45.330893 25382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
uuid: "038cb801e9214358bab57093f322c097"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:45.331243 25382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:45.353334 25382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:45.353811 25382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:45.353921 25382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:45.359881 25382 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.124:36421
I20240506 20:04:45.359936 25448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.124:36421 every 8 connection(s)
I20240506 20:04:45.360806 25382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
I20240506 20:04:45.364374 25449 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:04:45.365849 25449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap starting.
I20240506 20:04:45.366492 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 25382
I20240506 20:04:45.370937 25449 log.cc:824] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:45.378924 25449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=7 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 20:04:45.379218 25449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap complete.
I20240506 20:04:45.381685 25449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:45.381990 25449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038cb801e9214358bab57093f322c097, State: Initialized, Role: FOLLOWER
I20240506 20:04:45.382369 25449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:45.382742 25453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:45.382848 25453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:45.383239 25449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:04:45.383858 25454 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:45.383955 25454 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:45.384349 25454 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:04:45.384974 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:04:45.385064 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:45.385157 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:04:45.385288 25454 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:04:45.387482 25382 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:04:45.389037 25468 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:45.389101 25468 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: loading cluster ID for follower catalog manager: success
I20240506 20:04:45.389204 25382 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:45.391363 25468 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: acquiring CA information for follower catalog manager: success
I20240506 20:04:45.392709 25468 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:04:45.647956 24853 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:45.647997 24580 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:45.649324 25413 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:55995; Asking this server to re-register.
I20240506 20:04:45.649333 25412 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:40243; Asking this server to re-register.
I20240506 20:04:45.649782 24580 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:45.650233 24853 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:45.650825 25412 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:45.650930 25413 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:45.687106 24715 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:04:45.687790 25413 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:48883; Asking this server to re-register.
I20240506 20:04:45.688122 24715 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:45.688858 25413 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:45.966441 25474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:45.966637 25474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:45.967813 25474 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:45.968295 25398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:45.968902 25397 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:45.969015 25398 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:45.969110 25398 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:45.969329 25474 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 20:04:46.497160 25479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:46.497270 25479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:46.497650 25479 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:46.498723 25397 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:46.498818 25398 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:46.498893 25398 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:46.499027 25479 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 20:04:47.620224 25487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:47.620352 25487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:47.620764 25487 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:47.622045 25398 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:47.622154 25397 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:47.622220 25397 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:47.622344 25487 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 20:04:48.200954 25495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:48.201074 25495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:48.201439 25495 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:48.202488 25398 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:48.202579 25397 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:48.202654 25397 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:48.202781 25495 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 20:04:48.559255 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--webserver_interface=127.23.94.125
--webserver_port=43623
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:04:48.659071 25499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:48.659267 25499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:48.659310 25499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:48.662240 25499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:48.662285 25499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:48.662303 25499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:48.662321 25499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:48.665766 25499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.94.125
--webserver_port=43623
--never_fsync=true
--heap_profile_path=/tmp/kudu.25499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:48.666515 25499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:48.667492 25499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:48.673918 25507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:48.674311 25504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:48.674705 25505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:48.675189 25499 server_base.cc:1034] running on GCE node
I20240506 20:04:48.675643 25499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:48.676467 25499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:48.677624 25499 hybrid_clock.cc:648] HybridClock initialized: now 1715025888677597 us; error 39 us; skew 500 ppm
I20240506 20:04:48.679452 25499 webserver.cc:460] Webserver started at http://127.23.94.125:43623/ using document root <none> and password file <none>
I20240506 20:04:48.679948 25499 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:48.680001 25499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:48.683059 25499 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:48.684903 25515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:48.685982 25499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 20:04:48.686100 25499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
uuid: "853e921441fd4ee4b5bb84fdab584029"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:48.686529 25499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:48.704715 25499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:48.705236 25499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:48.705365 25499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:48.710934 25499 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.125:42097
I20240506 20:04:48.711091 25567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.125:42097 every 8 connection(s)
I20240506 20:04:48.711859 25499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
I20240506 20:04:48.713989 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 25499
I20240506 20:04:48.716348 25568 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:04:48.718014 25568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap starting.
I20240506 20:04:48.723151 25568 log.cc:824] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:48.731472 25568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=13 ignored=0} inserts{seen=8 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240506 20:04:48.731817 25568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap complete.
I20240506 20:04:48.734258 25568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:48.734710 25568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853e921441fd4ee4b5bb84fdab584029, State: Initialized, Role: FOLLOWER
I20240506 20:04:48.735121 25568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 5.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:48.735513 25572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:48.735622 25572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:48.736032 25568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:04:48.741762 25586 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:48.741864 25586 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: loading cluster ID for follower catalog manager: success
I20240506 20:04:48.740200 25499 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:04:48.742341 25499 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:48.744278 25586 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: acquiring CA information for follower catalog manager: success
I20240506 20:04:48.745693 25586 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:04:49.139024 24582 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:49.139098 24855 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:49.140393 25532 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:38743; Asking this server to re-register.
I20240506 20:04:49.140403 25529 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:43273; Asking this server to re-register.
I20240506 20:04:49.140861 24582 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:49.140858 24855 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:49.141917 25532 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:49.141994 25529 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:49.186278 24717 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:49.186997 25532 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:49621; Asking this server to re-register.
I20240506 20:04:49.187371 24717 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:49.188145 25532 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:49.275074 25595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:49.275272 25595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:49.276407 25595 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:49.277040 25517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:49.277846 25517 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:49.280174 25423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 6 candidate_status { last_received { term: 5 index: 14 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097" is_pre_election: true
I20240506 20:04:49.280402 25423 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853e921441fd4ee4b5bb84fdab584029 in term 5.
I20240506 20:04:49.280856 25517 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:49.281064 25595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 20:04:49.281128 25595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 20:04:49.281175 25595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 5 FOLLOWER]: Advancing to term 6
I20240506 20:04:49.282878 25595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:49.283205 25595 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 6 election: Requested vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:49.283656 25423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 6 candidate_status { last_received { term: 5 index: 14 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097"
I20240506 20:04:49.283756 25423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 5 FOLLOWER]: Advancing to term 6
W20240506 20:04:49.283965 25517 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:49.285360 25423 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853e921441fd4ee4b5bb84fdab584029 in term 6.
I20240506 20:04:49.285604 25517 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:49.285759 25595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 6 FOLLOWER]: Leader election won for term 6
I20240506 20:04:49.285933 25595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 6 LEADER]: Becoming Leader. State: Replica: 853e921441fd4ee4b5bb84fdab584029, State: Running, Role: LEADER
I20240506 20:04:49.286223 25595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 5.14, Last appended by leader: 14, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:49.287652 25600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 853e921441fd4ee4b5bb84fdab584029. Latest consensus state: current_term: 6 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:49.287756 25600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:49.288028 25602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:04:49.289479 25602 catalog_manager.cc:669] Loaded metadata for table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]
I20240506 20:04:49.290585 25602 tablet_loader.cc:96] loaded metadata for tablet af340b9943944cbb84a445e6c0034bc5 (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da])
I20240506 20:04:49.290704 25602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:04:49.291065 25602 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:49.291110 25602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:04:49.291596 25602 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:04:49.291970 25602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Loaded TSK: 0
I20240506 20:04:49.292553 25602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:04:49.292949 25602 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:04:49.293293 25602 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 11
I20240506 20:04:49.310045 25532 master_service.cc:946] ReplaceTablet: received request to replace tablet af340b9943944cbb84a445e6c0034bc5 from {username='slave'} at 127.0.0.1:45722
I20240506 20:04:49.310149 25531 master_service.cc:946] ReplaceTablet: received request to replace tablet af340b9943944cbb84a445e6c0034bc5 from {username='slave'} at 127.0.0.1:45722
I20240506 20:04:49.310272 25528 master_service.cc:946] ReplaceTablet: received request to replace tablet af340b9943944cbb84a445e6c0034bc5 from {username='slave'} at 127.0.0.1:45722
I20240506 20:04:49.310077 25529 master_service.cc:946] ReplaceTablet: received request to replace tablet af340b9943944cbb84a445e6c0034bc5 from {username='slave'} at 127.0.0.1:45722
I20240506 20:04:49.310840 25530 master_service.cc:946] ReplaceTablet: received request to replace tablet af340b9943944cbb84a445e6c0034bc5 from {username='slave'} at 127.0.0.1:45722
I20240506 20:04:49.310784 25524 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45722:
name: "default.Table_6d90713e02f146279b23811c97941079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:49.310971 25527 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45722:
name: "default.Table_e80158bfa50b427cbd790f160164b525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:49.310787 25526 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45722:
name: "default.Table_7f7588f0e29e4057a66f08687ec28975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:49.311208 25525 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45722:
name: "default.Table_8df5615aacee4940a5402206a921f0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:49.311373 25523 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45722:
name: "default.Table_6e4d3650620d474ab3107f0f53f5e807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:49.314147 25423 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Refusing update from remote peer 853e921441fd4ee4b5bb84fdab584029: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 6 index: 16. (index mismatch)
I20240506 20:04:49.314745 25600 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 10, Time since last communication: 0.000s
W20240506 20:04:49.315019 25517 consensus_peers.cc:485] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 -> Peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Couldn't send request to peer 25192b33b13d4b089dc3fb1d1f6fc2e9. Status: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 20:04:49.317169 25585 catalog_manager.cc:2706] Deleting table default.table_c568fe36c59c4d93bda72f412845b2f2 [id=f8c2559276c74cc698dc2e82a2d7bf5f] in response to Hive Metastore notification log event 12
W20240506 20:04:49.317442 25585 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_c568fe36c59c4d93bda72f412845b2f2: Not found: the table does not exist: table_id: "f8c2559276c74cc698dc2e82a2d7bf5f" table_name: "default.table_c568fe36c59c4d93bda72f412845b2f2"
F20240506 20:04:49.317495 25585 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 12): 12 CREATE_TABLE default.table_c66444cb675241d1a23c4348383b208e
*** Check failure stack trace: ***
I20240506 20:04:49.319686 25604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 853e921441fd4ee4b5bb84fdab584029. Latest consensus state: current_term: 6 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:49.319795 25604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:49.319774 25607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:49.319837 25607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f2e1324bdcd google::LogMessage::Fail() at ??:0
I20240506 20:04:49.321296 25605 mvcc.cc:205] Tried to move back new op lower bound from 7024746042625175552 to 7024745990614958080. Current Snapshot: MvccSnapshot[applied={T|T < 7024746042625175552}]
I20240506 20:04:49.321276 25607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:49.321385 25607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:49.321466 25604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:49.321529 25604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f2e1324fb93 google::LogMessage::SendToLog() at ??:0
I20240506 20:04:49.322317 25595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:49.322414 25595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:49.322374 25600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:49.322464 25600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:49.322964 25603 mvcc.cc:205] Tried to move back new op lower bound from 7024746042625175552 to 7024746009559257088. Current Snapshot: MvccSnapshot[applied={T|T < 7024746042625175552}]
@ 0x7f2e1324b7cc google::LogMessage::Flush() at ??:0
I20240506 20:04:49.324513 25528 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45722:
name: "default.Table_a1b4d3a0d5ce479584c8f895844cadc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:49.324513 25531 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45722:
name: "default.Table_638fa9e2f7b349a38f86c3d585998fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:49.324564 25530 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45722:
name: "default.Table_758d3e4f504a489eb12b7bf93d583cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f2e1324ba11 google::LogMessage::~LogMessage() at ??:0
I20240506 20:04:49.324926 25529 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45722:
name: "default.Table_2ee32586537f466fa07ee5ad6c48dec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:49.325225 25532 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Sending DeleteTablet for 3 replicas of tablet af340b9943944cbb84a445e6c0034bc5
I20240506 20:04:49.326234 25532 catalog_manager.cc:6554] ReplaceTablet: tablet af340b9943944cbb84a445e6c0034bc5 deleted and replaced by tablet 76234ea981fe4ae4a41171ed2586e27e
I20240506 20:04:49.326480 25532 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45722:
table { table_name: "default.Table_c66444cb675241d1a23c4348383b208e" } new_table_name: "default.Table_ea33dccc2852419e87b57fc573f6572e" modify_external_catalogs: true
@ 0x7f2e181774f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240506 20:04:49.334002 24514 tablet_service.cc:1514] Processing DeleteTablet for tablet af340b9943944cbb84a445e6c0034bc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 76234ea981fe4ae4a41171ed2586e27e) from {username='slave'} at 127.0.0.1:59860
@ 0x7f2e181765ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240506 20:04:49.334601 24513 tablet_service.cc:1467] Processing CreateTablet for tablet 76234ea981fe4ae4a41171ed2586e27e (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:49.334884 24513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76234ea981fe4ae4a41171ed2586e27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:49.336285 24787 tablet_service.cc:1467] Processing CreateTablet for tablet 76234ea981fe4ae4a41171ed2586e27e (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:49.336508 24787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76234ea981fe4ae4a41171ed2586e27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:49.337375 24786 tablet_service.cc:1514] Processing DeleteTablet for tablet af340b9943944cbb84a445e6c0034bc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 76234ea981fe4ae4a41171ed2586e27e) from {username='slave'} at 127.0.0.1:43274
I20240506 20:04:49.337648 25622 tablet_replica.cc:330] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: stopping tablet replica
I20240506 20:04:49.337754 25622 raft_consensus.cc:2229] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 20:04:49.337870 25622 raft_consensus.cc:2258] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:49.338272 25622 ts_tablet_manager.cc:1875] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:49.338584 24648 tablet_service.cc:1514] Processing DeleteTablet for tablet af340b9943944cbb84a445e6c0034bc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 76234ea981fe4ae4a41171ed2586e27e) from {username='slave'} at 127.0.0.1:40110
I20240506 20:04:49.338884 25624 tablet_replica.cc:330] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: stopping tablet replica
I20240506 20:04:49.338997 25624 raft_consensus.cc:2229] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 1 LEADER]: Raft consensus shutting down.
I20240506 20:04:49.338973 24649 tablet_service.cc:1467] Processing CreateTablet for tablet 76234ea981fe4ae4a41171ed2586e27e (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f2e18175d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20240506 20:04:49.339215 24649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76234ea981fe4ae4a41171ed2586e27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:49.339293 25624 raft_consensus.cc:2258] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:49.339664 25624 ts_tablet_manager.cc:1875] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:49.340471 25622 ts_tablet_manager.cc:1888] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 20:04:49.340525 25622 log.cc:1196] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal/wals/af340b9943944cbb84a445e6c0034bc5
I20240506 20:04:49.340798 25622 ts_tablet_manager.cc:1909] T af340b9943944cbb84a445e6c0034bc5 P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting consensus metadata
I20240506 20:04:49.341631 25624 ts_tablet_manager.cc:1888] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 20:04:49.341692 25624 log.cc:1196] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal/wals/af340b9943944cbb84a445e6c0034bc5
I20240506 20:04:49.341682 25516 catalog_manager.cc:4830] TS b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): tablet af340b9943944cbb84a445e6c0034bc5 (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
I20240506 20:04:49.341912 25624 ts_tablet_manager.cc:1909] T af340b9943944cbb84a445e6c0034bc5 P c35a1e4b657846ba945aa2602bea72cb: Deleting consensus metadata
I20240506 20:04:49.342497 25621 tablet_bootstrap.cc:492] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: Bootstrap starting.
I20240506 20:04:49.342710 25518 catalog_manager.cc:4830] TS c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): tablet af340b9943944cbb84a445e6c0034bc5 (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
I20240506 20:04:49.343508 25621 tablet_bootstrap.cc:654] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: Neither blocks nor log segments found. Creating new log.
@ 0x7f2e1817afc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240506 20:04:49.344843 25621 tablet_bootstrap.cc:492] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: No bootstrap required, opened a new log
I20240506 20:04:49.344930 25621 ts_tablet_manager.cc:1367] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
@ 0x5569610ce75c std::function<>::operator()() at ??:0
I20240506 20:04:49.345494 25621 raft_consensus.cc:348] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:49.345597 25621 raft_consensus.cc:374] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:49.345624 25621 raft_consensus.cc:729] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec4b56ff88545a69ab015a9d695d5d9, State: Initialized, Role: FOLLOWER
I20240506 20:04:49.345746 25621 consensus_queue.cc:260] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:49.346022 25621 ts_tablet_manager.cc:1398] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7f2e1468c807 kudu::Thread::SuperviseThread() at ??:0
I20240506 20:04:49.348783 25620 tablet_replica.cc:330] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: stopping tablet replica
I20240506 20:04:49.348804 25623 tablet_bootstrap.cc:492] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: Bootstrap starting.
I20240506 20:04:49.348867 25620 raft_consensus.cc:2229] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 20:04:49.348942 25620 raft_consensus.cc:2258] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:49.349197 25620 ts_tablet_manager.cc:1875] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:49.349521 25623 tablet_bootstrap.cc:654] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:49.350207 25625 tablet_bootstrap.cc:492] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: Bootstrap starting.
@ 0x7f2e14eca6db start_thread at ??:0
I20240506 20:04:49.351027 25620 ts_tablet_manager.cc:1888] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 20:04:49.351089 25620 log.cc:1196] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal/wals/af340b9943944cbb84a445e6c0034bc5
I20240506 20:04:49.351109 25625 tablet_bootstrap.cc:654] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:49.351336 25620 ts_tablet_manager.cc:1909] T af340b9943944cbb84a445e6c0034bc5 P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting consensus metadata
@ 0x7f2e1240271f clone at ??:0
I20240506 20:04:49.352795 25625 tablet_bootstrap.cc:492] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: No bootstrap required, opened a new log
I20240506 20:04:49.352849 25623 tablet_bootstrap.cc:492] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: No bootstrap required, opened a new log
I20240506 20:04:49.352876 25625 ts_tablet_manager.cc:1367] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240506 20:04:49.352916 25623 ts_tablet_manager.cc:1367] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240506 20:04:49.353309 25623 raft_consensus.cc:348] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:49.353309 25625 raft_consensus.cc:348] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:49.353410 25623 raft_consensus.cc:374] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:49.353416 25625 raft_consensus.cc:374] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:49.353440 25623 raft_consensus.cc:729] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ecd08373ba4b2cb8623b3f4862bb98, State: Initialized, Role: FOLLOWER
I20240506 20:04:49.353446 25625 raft_consensus.cc:729] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35a1e4b657846ba945aa2602bea72cb, State: Initialized, Role: FOLLOWER
I20240506 20:04:49.353569 25625 consensus_queue.cc:260] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:49.353569 25623 consensus_queue.cc:260] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:49.353824 25625 ts_tablet_manager.cc:1398] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 20:04:49.353891 25623 ts_tablet_manager.cc:1398] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240506 20:04:49.494807 24470 connection.cc:535] server connection from 127.0.0.1:59860 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:49.494815 24602 connection.cc:535] client connection to 127.23.94.125:42097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:49.495222 24740 connection.cc:535] client connection to 127.23.94.125:42097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:49.495573 24860 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.94.125:42097)
W20240506 20:04:49.495718 24860 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.23.94.125:42097)
W20240506 20:04:49.526861 24860 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.23.94.125:42097)
I20240506 20:04:49.615455 25119 raft_consensus.cc:482] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:49.615571 25119 raft_consensus.cc:504] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:49.615960 25119 leader_election.cc:288] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:49.616308 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76234ea981fe4ae4a41171ed2586e27e" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" is_pre_election: true
I20240506 20:04:49.616345 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76234ea981fe4ae4a41171ed2586e27e" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" is_pre_election: true
I20240506 20:04:49.616436 24807 raft_consensus.cc:2454] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 0.
I20240506 20:04:49.616487 24534 raft_consensus.cc:2454] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 0.
I20240506 20:04:49.616773 24602 leader_election.cc:302] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb; no voters:
I20240506 20:04:49.616941 25119 raft_consensus.cc:2781] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 20:04:49.616998 25119 raft_consensus.cc:482] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 20:04:49.617043 25119 raft_consensus.cc:3037] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:49.617870 25119 raft_consensus.cc:504] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:49.618165 25119 leader_election.cc:288] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 election: Requested vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:49.618460 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76234ea981fe4ae4a41171ed2586e27e" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98"
I20240506 20:04:49.618463 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76234ea981fe4ae4a41171ed2586e27e" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec4b56ff88545a69ab015a9d695d5d9"
I20240506 20:04:49.618556 24534 raft_consensus.cc:3037] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:49.618561 24807 raft_consensus.cc:3037] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:49.619268 24807 raft_consensus.cc:2454] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 1.
I20240506 20:04:49.619325 24534 raft_consensus.cc:2454] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 1.
I20240506 20:04:49.619553 24602 leader_election.cc:302] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb; no voters:
I20240506 20:04:49.619747 25119 raft_consensus.cc:2781] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Leader election won for term 1
I20240506 20:04:49.619935 25119 raft_consensus.cc:686] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 1 LEADER]: Becoming Leader. State: Replica: c35a1e4b657846ba945aa2602bea72cb, State: Running, Role: LEADER
I20240506 20:04:49.620088 25119 consensus_queue.cc:237] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:49.832088 25644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:49.832229 25644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:49.832602 25644 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:49.833559 25398 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:49.833668 25397 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:49.833745 25397 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:49.833984 25644 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 20:04:50.241168 25650 consensus_queue.cc:1035] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 20:04:50.243741 25650 consensus_queue.cc:1035] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 20:04:50.390116 25382 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:04:50.390246 25382 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:50.554287 25664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:50.554395 25664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:50.554761 25664 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:50.555732 25398 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:50.555838 25397 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:50.555923 25397 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:50.556046 25664 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 20:04:51.136055 25672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:51.136183 25672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:51.136595 25672 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:51.137145 25398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240506 20:04:51.137802 25397 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:51.137882 25398 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:04:51.137959 25398 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:51.138077 25672 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 20:04:51.769990 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.126:40141
--webserver_interface=127.23.94.126
--webserver_port=45265
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:04:51.871534 25689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:51.871752 25689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:51.871805 25689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:51.874965 25689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:51.875020 25689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:51.875046 25689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:51.875069 25689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:51.878569 25689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.126:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.94.126
--webserver_port=45265
--never_fsync=true
--heap_profile_path=/tmp/kudu.25689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:51.879350 25689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:51.880362 25689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:51.887096 25694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:51.887099 25695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:51.887513 25697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:51.887952 25689 server_base.cc:1034] running on GCE node
I20240506 20:04:51.888365 25689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:51.889231 25689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:51.890372 25689 hybrid_clock.cc:648] HybridClock initialized: now 1715025891890360 us; error 23 us; skew 500 ppm
I20240506 20:04:51.897430 25689 webserver.cc:460] Webserver started at http://127.23.94.126:45265/ using document root <none> and password file <none>
I20240506 20:04:51.897964 25689 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:51.898029 25689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:51.901269 25689 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240506 20:04:51.903084 25703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:51.904019 25689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:51.904131 25689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:51.904534 25689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:51.939893 25689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:51.940546 25689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:51.940734 25689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:51.947450 25689 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.126:40141
I20240506 20:04:51.947456 25755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.126:40141 every 8 connection(s)
I20240506 20:04:51.948436 25689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/info.pb
I20240506 20:04:51.952257 25756 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:04:51.953769 25756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Bootstrap starting.
I20240506 20:04:51.955297 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 25689
I20240506 20:04:51.959973 25756 log.cc:824] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:51.968894 25756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=8 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 20:04:51.969220 25756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Bootstrap complete.
I20240506 20:04:51.971814 25756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:51.972218 25756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25192b33b13d4b089dc3fb1d1f6fc2e9, State: Initialized, Role: FOLLOWER
I20240506 20:04:51.972671 25756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 4.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:51.973068 25760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:51.973186 25760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:51.973645 25756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:04:51.978969 25774 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:51.979053 25774 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: loading cluster ID for follower catalog manager: success
I20240506 20:04:51.980899 25774 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: acquiring CA information for follower catalog manager: success
I20240506 20:04:51.982254 25774 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:04:51.983748 25689 master_runner.cc:186] Error getting master registration for 127.23.94.125:42097: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111),
I20240506 20:04:51.985863 25689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:52.081949 25778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:52.082108 25778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:52.082568 25778 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:04:52.084409 25397 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:52.087538 25725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 7 candidate_status { last_received { term: 6 index: 17 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" is_pre_election: true
I20240506 20:04:52.087829 25725 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038cb801e9214358bab57093f322c097 in term 4.
I20240506 20:04:52.088238 25398 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:52.088460 25778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 20:04:52.088510 25778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:52.088546 25778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 6 FOLLOWER]: Advancing to term 7
I20240506 20:04:52.089407 25778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:52.089769 25778 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 election: Requested vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
I20240506 20:04:52.090181 25725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 7 candidate_status { last_received { term: 6 index: 17 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
I20240506 20:04:52.090299 25725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 4 FOLLOWER]: Advancing to term 7
W20240506 20:04:52.090665 25397 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:52.092011 25725 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038cb801e9214358bab57093f322c097 in term 7.
I20240506 20:04:52.092284 25398 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:52.092486 25778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 7 FOLLOWER]: Leader election won for term 7
I20240506 20:04:52.092685 25778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 7 LEADER]: Becoming Leader. State: Replica: 038cb801e9214358bab57093f322c097, State: Running, Role: LEADER
I20240506 20:04:52.092944 25778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 6.17, Last appended by leader: 17, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:52.093828 25782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 038cb801e9214358bab57093f322c097. Latest consensus state: current_term: 7 leader_uuid: "038cb801e9214358bab57093f322c097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:52.093919 25782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:52.094174 25784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:04:52.095526 25784 catalog_manager.cc:669] Loaded metadata for table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]
I20240506 20:04:52.096606 25784 tablet_loader.cc:96] loaded metadata for tablet 76234ea981fe4ae4a41171ed2586e27e (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da])
I20240506 20:04:52.096923 25784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:04:52.097321 25784 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:52.097363 25784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:04:52.097872 25784 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:04:52.098251 25784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Loaded TSK: 0
I20240506 20:04:52.098811 25784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:04:52.099195 25784 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:04:52.099545 25784 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 11
I20240506 20:04:52.120941 25785 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:52.121048 25785 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:52.121505 25785 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:04:52.122165 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:04:52.122339 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:52.122395 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:04:52.122530 25785 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:04:52.161310 25413 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38822:
table { table_name: "default.Table_c66444cb675241d1a23c4348383b208e" } new_table_name: "default.Table_ea33dccc2852419e87b57fc573f6572e" modify_external_catalogs: true
I20240506 20:04:52.161389 25411 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38822:
name: "default.Table_2ee32586537f466fa07ee5ad6c48dec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:52.161521 25408 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38822:
name: "default.Table_758d3e4f504a489eb12b7bf93d583cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:52.161345 25412 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38822:
name: "default.Table_c568fe36c59c4d93bda72f412845b2f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:52.161835 25409 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38822:
name: "default.Table_a1b4d3a0d5ce479584c8f895844cadc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:52.161871 25404 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38822:
name: "default.Table_7f7588f0e29e4057a66f08687ec28975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:52.162058 25407 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38822:
name: "default.Table_6e4d3650620d474ab3107f0f53f5e807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:52.162092 25410 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38822:
name: "default.Table_638fa9e2f7b349a38f86c3d585998fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:52.162137 25405 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38822:
name: "default.Table_8df5615aacee4940a5402206a921f0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:52.162303 25406 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38822:
name: "default.Table_6d90713e02f146279b23811c97941079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:52.166733 25467 catalog_manager.cc:2706] Deleting table default.table_c568fe36c59c4d93bda72f412845b2f2 [id=f8c2559276c74cc698dc2e82a2d7bf5f] in response to Hive Metastore notification log event 12
W20240506 20:04:52.167002 25467 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_c568fe36c59c4d93bda72f412845b2f2: Not found: the table does not exist: table_id: "f8c2559276c74cc698dc2e82a2d7bf5f" table_name: "default.table_c568fe36c59c4d93bda72f412845b2f2"
F20240506 20:04:52.167062 25467 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 12): 12 CREATE_TABLE default.table_c66444cb675241d1a23c4348383b208e
*** Check failure stack trace: ***
@ 0x7fd3c94aedcd google::LogMessage::Fail() at ??:0
@ 0x7fd3c94b2b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd3c94ae7cc google::LogMessage::Flush() at ??:0
@ 0x7fd3c94aea11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fd3ce3da4f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fd3ce3d95ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fd3ce3d8d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7fd3ce3ddfc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x560c1078675c std::function<>::operator()() at ??:0
@ 0x7fd3ca8ef807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd3cb12d6db start_thread at ??:0
@ 0x7fd3c866571f clone at ??:0
I20240506 20:04:52.228581 24716 heartbeater.cc:347] Connected to a master server at 127.23.94.126:40141
I20240506 20:04:52.228752 24716 heartbeater.cc:503] Master 127.23.94.126:40141 was elected leader, sending a full tablet report...
I20240506 20:04:52.230319 25718 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:40425; Asking this server to re-register.
I20240506 20:04:52.230787 24716 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:52.231773 25718 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:52.252275 24581 heartbeater.cc:347] Connected to a master server at 127.23.94.126:40141
I20240506 20:04:52.253319 25718 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:33837; Asking this server to re-register.
I20240506 20:04:52.253639 24854 heartbeater.cc:347] Connected to a master server at 127.23.94.126:40141
I20240506 20:04:52.253793 24854 heartbeater.cc:503] Master 127.23.94.126:40141 was elected leader, sending a full tablet report...
I20240506 20:04:52.254307 24581 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:52.255036 25718 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:33645; Asking this server to re-register.
I20240506 20:04:52.255215 25714 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:52.255571 24854 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:52.256459 25714 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
W20240506 20:04:52.339226 24741 connection.cc:535] client connection to 127.23.94.124:36421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:52.339221 24603 connection.cc:535] client connection to 127.23.94.124:36421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:52.339629 24468 connection.cc:535] client connection to 127.23.94.124:36421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:04:52.340296 24861 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.23.94.124:36421)
W20240506 20:04:52.340667 24861 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.94.124:36421)
W20240506 20:04:52.340952 24861 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.23.94.124:36421)
I20240506 20:04:52.657253 25805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:52.657383 25805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:52.658496 25805 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:52.658982 25704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:52.659651 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
W20240506 20:04:52.659780 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:52.659847 25704 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:52.660038 25805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 20:04:53.283006 25809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:53.283124 25809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:53.283499 25809 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:53.284523 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:53.284606 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:53.284694 25705 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:53.284817 25809 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 20:04:53.937872 25813 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:53.938004 25813 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:04:53.938445 25813 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:04:53.939186 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:04:53.939301 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:04:53.939360 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:04:53.939496 25813 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:04:54.011210 25814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:54.011317 25814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:54.011673 25814 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:54.012579 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:54.012699 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:54.012760 25705 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:54.012877 25814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 20:04:55.051066 25818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:55.051188 25818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:55.051570 25818 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:55.052579 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:55.052692 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:55.052770 25705 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:55.052891 25818 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 20:04:55.117966 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--webserver_interface=127.23.94.125
--webserver_port=43623
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:04:55.219570 25822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:55.219771 25822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:55.219815 25822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:55.223018 25822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:55.223078 25822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:55.223096 25822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:55.223115 25822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:55.226687 25822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.94.125
--webserver_port=43623
--never_fsync=true
--heap_profile_path=/tmp/kudu.25822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:55.227488 25822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:55.228511 25822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:55.235558 25829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:55.235673 25828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:55.236050 25831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:55.236481 25822 server_base.cc:1034] running on GCE node
I20240506 20:04:55.236927 25822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:55.237725 25822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:55.238867 25822 hybrid_clock.cc:648] HybridClock initialized: now 1715025895238858 us; error 20 us; skew 500 ppm
I20240506 20:04:55.240490 25822 webserver.cc:460] Webserver started at http://127.23.94.125:43623/ using document root <none> and password file <none>
I20240506 20:04:55.241008 25822 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:55.241066 25822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:55.244102 25822 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240506 20:04:55.245958 25837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:55.246850 25822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 20:04:55.246956 25822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
uuid: "853e921441fd4ee4b5bb84fdab584029"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:55.247323 25822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:55.280575 25822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:55.281212 25822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:55.281360 25822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:55.287569 25822 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.125:42097
I20240506 20:04:55.287567 25889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.125:42097 every 8 connection(s)
I20240506 20:04:55.288558 25822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
I20240506 20:04:55.292380 25890 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:04:55.292984 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 25822
I20240506 20:04:55.294065 25890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap starting.
I20240506 20:04:55.300547 25890 log.cc:824] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:55.310596 25890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=9 ignored=0} mutations{seen=7 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 20:04:55.310887 25890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap complete.
I20240506 20:04:55.313341 25890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:55.313966 25890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853e921441fd4ee4b5bb84fdab584029, State: Initialized, Role: FOLLOWER
I20240506 20:04:55.314360 25890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 6.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:55.314718 25895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:55.314827 25895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:55.315251 25890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:04:55.320407 25909 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:55.320513 25909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: loading cluster ID for follower catalog manager: success
I20240506 20:04:55.322831 25909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: acquiring CA information for follower catalog manager: success
I20240506 20:04:55.324254 25909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:04:55.334316 25822 master_runner.cc:186] Error getting master registration for 127.23.94.124:36421: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111),
I20240506 20:04:55.336124 25822 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:55.644726 25914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:55.644848 25914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:55.645265 25914 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:55.646391 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:55.648653 25864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 8 candidate_status { last_received { term: 4 index: 13 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029" is_pre_election: true
I20240506 20:04:55.648927 25864 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 for term 8 because replica has last-logged OpId of term: 6 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 13.
I20240506 20:04:55.649328 25704 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:55.649479 25914 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240506 20:04:55.734321 24717 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:55.735775 25854 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:55783; Asking this server to re-register.
I20240506 20:04:55.736207 24717 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:55.737183 25854 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:55.755260 24582 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:55.756063 25854 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:56593; Asking this server to re-register.
I20240506 20:04:55.756407 24582 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:55.757139 25854 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:55.758531 24855 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:55.759235 25854 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:56865; Asking this server to re-register.
I20240506 20:04:55.759578 24855 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:55.760233 25854 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:55.827231 25920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:55.827435 25920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:55.828521 25920 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:55.828933 25725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 7 candidate_status { last_received { term: 6 index: 17 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" is_pre_election: true
I20240506 20:04:55.829119 25725 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 853e921441fd4ee4b5bb84fdab584029 in current term 7: Already voted for candidate 038cb801e9214358bab57093f322c097 in this term.
W20240506 20:04:55.829088 25839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
W20240506 20:04:55.829841 25839 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:55.829913 25839 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:55.830116 25920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 6 FOLLOWER]: Advancing to term 7
I20240506 20:04:55.831885 25920 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240506 20:04:56.364198 25924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:56.364318 25924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:56.364765 25924 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:56.365164 25725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 8 candidate_status { last_received { term: 6 index: 17 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" is_pre_election: true
I20240506 20:04:56.365304 25725 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853e921441fd4ee4b5bb84fdab584029 in term 7.
I20240506 20:04:56.365617 25839 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029; no voters:
I20240506 20:04:56.365794 25924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 20:04:56.365870 25924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 20:04:56.365922 25924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 7 FOLLOWER]: Advancing to term 8
W20240506 20:04:56.366106 25839 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:56.366832 25924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:56.367121 25924 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 8 election: Requested vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:56.367434 25725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 8 candidate_status { last_received { term: 6 index: 17 } } ignore_live_leader: false dest_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
I20240506 20:04:56.367554 25725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 7 FOLLOWER]: Advancing to term 8
W20240506 20:04:56.368008 25839 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:56.368438 25725 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853e921441fd4ee4b5bb84fdab584029 in term 8.
I20240506 20:04:56.368712 25839 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097
I20240506 20:04:56.368852 25924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 8 FOLLOWER]: Leader election won for term 8
I20240506 20:04:56.368990 25924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 8 LEADER]: Becoming Leader. State: Replica: 853e921441fd4ee4b5bb84fdab584029, State: Running, Role: LEADER
I20240506 20:04:56.369282 25924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 6.17, Last appended by leader: 17, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:56.370811 25927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 853e921441fd4ee4b5bb84fdab584029. Latest consensus state: current_term: 8 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:56.370899 25927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:56.371156 25929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:04:56.372526 25929 catalog_manager.cc:669] Loaded metadata for table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]
I20240506 20:04:56.373667 25929 tablet_loader.cc:96] loaded metadata for tablet 76234ea981fe4ae4a41171ed2586e27e (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da])
I20240506 20:04:56.373886 25929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:04:56.374289 25929 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:56.374332 25929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:04:56.374888 25929 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:04:56.375262 25929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Loaded TSK: 0
I20240506 20:04:56.375907 25929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:04:56.376291 25929 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:04:56.376672 25929 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 11
I20240506 20:04:56.394157 25854 master_service.cc:946] ReplaceTablet: received request to replace tablet 76234ea981fe4ae4a41171ed2586e27e from {username='slave'} at 127.0.0.1:52706
I20240506 20:04:56.394197 25853 master_service.cc:946] ReplaceTablet: received request to replace tablet 76234ea981fe4ae4a41171ed2586e27e from {username='slave'} at 127.0.0.1:52706
I20240506 20:04:56.394418 25851 master_service.cc:946] ReplaceTablet: received request to replace tablet 76234ea981fe4ae4a41171ed2586e27e from {username='slave'} at 127.0.0.1:52706
I20240506 20:04:56.394760 25849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "default.Table_e80158bfa50b427cbd790f160164b525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:56.394893 25847 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "default.Table_6d90713e02f146279b23811c97941079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:56.395087 25850 master_service.cc:946] ReplaceTablet: received request to replace tablet 76234ea981fe4ae4a41171ed2586e27e from {username='slave'} at 127.0.0.1:52706
I20240506 20:04:56.394771 25846 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "default.Table_8df5615aacee4940a5402206a921f0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:56.395054 25852 master_service.cc:946] ReplaceTablet: received request to replace tablet 76234ea981fe4ae4a41171ed2586e27e from {username='slave'} at 127.0.0.1:52706
I20240506 20:04:56.395316 25845 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "default.Table_6e4d3650620d474ab3107f0f53f5e807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:56.395570 25848 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "default.Table_7f7588f0e29e4057a66f08687ec28975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:56.397984 25725 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Refusing update from remote peer 853e921441fd4ee4b5bb84fdab584029: Log matching property violated. Preceding OpId in replica: term: 4 index: 13. Preceding OpId from leader: term: 8 index: 19. (index mismatch)
W20240506 20:04:56.398432 25839 consensus_peers.cc:485] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 -> Peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Couldn't send request to peer 038cb801e9214358bab57093f322c097. Status: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 20:04:56.398684 25924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 13, Time since last communication: 0.000s
I20240506 20:04:56.400918 25908 catalog_manager.cc:2706] Deleting table default.table_c568fe36c59c4d93bda72f412845b2f2 [id=f8c2559276c74cc698dc2e82a2d7bf5f] in response to Hive Metastore notification log event 12
W20240506 20:04:56.401165 25908 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_c568fe36c59c4d93bda72f412845b2f2: Not found: the table does not exist: table_id: "f8c2559276c74cc698dc2e82a2d7bf5f" table_name: "default.table_c568fe36c59c4d93bda72f412845b2f2"
F20240506 20:04:56.401218 25908 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 12): 12 CREATE_TABLE default.table_c66444cb675241d1a23c4348383b208e
*** Check failure stack trace: ***
I20240506 20:04:56.403100 25931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 853e921441fd4ee4b5bb84fdab584029. Latest consensus state: current_term: 8 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:56.403195 25931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:56.403302 25934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:56.403348 25935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:56.403394 25934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:56.403412 25935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f5c241fadcd google::LogMessage::Fail() at ??:0
I20240506 20:04:56.404362 25930 mvcc.cc:205] Tried to move back new op lower bound from 7024746071639752704 to 7024746071531048960. Current Snapshot: MvccSnapshot[applied={T|T < 7024746071639752704}]
I20240506 20:04:56.404393 25932 mvcc.cc:205] Tried to move back new op lower bound from 7024746042690252800 to 7024746009559257088. Current Snapshot: MvccSnapshot[applied={T|T < 7024746042690252800 or (T in {7024746042690252800})}]
I20240506 20:04:56.405042 25935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:56.405130 25935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:56.404464 25927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:56.405504 25927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: LEADER
@ 0x7f5c241feb93 google::LogMessage::SendToLog() at ??:0
I20240506 20:04:56.406029 25854 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Sending DeleteTablet for 3 replicas of tablet 76234ea981fe4ae4a41171ed2586e27e
I20240506 20:04:56.406456 25850 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52706:
table { table_name: "default.Table_c66444cb675241d1a23c4348383b208e" } new_table_name: "default.Table_ea33dccc2852419e87b57fc573f6572e" modify_external_catalogs: true
@ 0x7f5c241fa7cc google::LogMessage::Flush() at ??:0
I20240506 20:04:56.407512 25853 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "default.Table_c568fe36c59c4d93bda72f412845b2f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:56.407909 25851 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "default.Table_2ee32586537f466fa07ee5ad6c48dec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f5c241faa11 google::LogMessage::~LogMessage() at ??:0
I20240506 20:04:56.408605 25852 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "default.Table_638fa9e2f7b349a38f86c3d585998fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:56.412091 25854 catalog_manager.cc:6554] ReplaceTablet: tablet 76234ea981fe4ae4a41171ed2586e27e deleted and replaced by tablet d86be6033bc84646ae3895c2c1242042
I20240506 20:04:56.412308 25854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "default.Table_a1b4d3a0d5ce479584c8f895844cadc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f5c291264f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240506 20:04:56.414947 24787 tablet_service.cc:1514] Processing DeleteTablet for tablet 76234ea981fe4ae4a41171ed2586e27e with delete_type TABLET_DATA_DELETED (replaced by tablet d86be6033bc84646ae3895c2c1242042) from {username='slave'} at 127.0.0.1:55854
I20240506 20:04:56.415445 24513 tablet_service.cc:1514] Processing DeleteTablet for tablet 76234ea981fe4ae4a41171ed2586e27e with delete_type TABLET_DATA_DELETED (replaced by tablet d86be6033bc84646ae3895c2c1242042) from {username='slave'} at 127.0.0.1:49040
I20240506 20:04:56.415736 25947 tablet_replica.cc:330] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: stopping tablet replica
I20240506 20:04:56.415846 25947 raft_consensus.cc:2229] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 20:04:56.415947 25947 raft_consensus.cc:2258] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:56.416364 25947 ts_tablet_manager.cc:1875] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:56.416638 24787 tablet_service.cc:1467] Processing CreateTablet for tablet d86be6033bc84646ae3895c2c1242042 (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:56.416857 24513 tablet_service.cc:1467] Processing CreateTablet for tablet d86be6033bc84646ae3895c2c1242042 (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:56.416904 24787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86be6033bc84646ae3895c2c1242042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:56.417158 24513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86be6033bc84646ae3895c2c1242042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:56.418583 25948 tablet_bootstrap.cc:492] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: Bootstrap starting.
I20240506 20:04:56.418885 25947 ts_tablet_manager.cc:1888] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 20:04:56.418884 24649 tablet_service.cc:1514] Processing DeleteTablet for tablet 76234ea981fe4ae4a41171ed2586e27e with delete_type TABLET_DATA_DELETED (replaced by tablet d86be6033bc84646ae3895c2c1242042) from {username='slave'} at 127.0.0.1:46304
I20240506 20:04:56.418946 25947 log.cc:1196] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal/wals/76234ea981fe4ae4a41171ed2586e27e
I20240506 20:04:56.419189 25947 ts_tablet_manager.cc:1909] T 76234ea981fe4ae4a41171ed2586e27e P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting consensus metadata
I20240506 20:04:56.419248 24648 tablet_service.cc:1467] Processing CreateTablet for tablet d86be6033bc84646ae3895c2c1242042 (DEFAULT_TABLE table=default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 20:04:56.419466 24648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86be6033bc84646ae3895c2c1242042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 20:04:56.419531 25948 tablet_bootstrap.cc:654] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:56.419737 25949 tablet_bootstrap.cc:492] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: Bootstrap starting.
I20240506 20:04:56.419750 25840 catalog_manager.cc:4830] TS 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): tablet 76234ea981fe4ae4a41171ed2586e27e (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
I20240506 20:04:56.420467 25949 tablet_bootstrap.cc:654] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:56.420557 25948 tablet_bootstrap.cc:492] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: No bootstrap required, opened a new log
I20240506 20:04:56.420650 25948 ts_tablet_manager.cc:1367] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:56.421072 25948 raft_consensus.cc:348] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:56.421159 25948 raft_consensus.cc:374] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:56.421186 25948 raft_consensus.cc:729] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ecd08373ba4b2cb8623b3f4862bb98, State: Initialized, Role: FOLLOWER
I20240506 20:04:56.421295 25949 tablet_bootstrap.cc:492] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: No bootstrap required, opened a new log
I20240506 20:04:56.421299 25948 consensus_queue.cc:260] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:56.421365 25949 ts_tablet_manager.cc:1367] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240506 20:04:56.421388 25953 tablet_bootstrap.cc:492] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb: Bootstrap starting.
I20240506 20:04:56.421651 25948 ts_tablet_manager.cc:1398] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240506 20:04:56.421758 25949 raft_consensus.cc:348] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:56.421847 25949 raft_consensus.cc:374] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:56.421876 25949 raft_consensus.cc:729] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec4b56ff88545a69ab015a9d695d5d9, State: Initialized, Role: FOLLOWER
I20240506 20:04:56.421977 25949 consensus_queue.cc:260] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:56.422814 25953 tablet_bootstrap.cc:654] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb: Neither blocks nor log segments found. Creating new log.
I20240506 20:04:56.423571 25949 ts_tablet_manager.cc:1398] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
@ 0x7f5c291255ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240506 20:04:56.425410 25953 tablet_bootstrap.cc:492] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb: No bootstrap required, opened a new log
I20240506 20:04:56.425493 25953 ts_tablet_manager.cc:1367] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240506 20:04:56.425882 25953 raft_consensus.cc:348] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:56.425954 25953 raft_consensus.cc:374] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 20:04:56.425973 25953 raft_consensus.cc:729] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35a1e4b657846ba945aa2602bea72cb, State: Initialized, Role: FOLLOWER
I20240506 20:04:56.426064 25953 consensus_queue.cc:260] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:56.426255 25953 ts_tablet_manager.cc:1398] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
@ 0x7f5c29124d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7f5c29129fc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240506 20:04:56.428930 25950 tablet_replica.cc:330] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: stopping tablet replica
I20240506 20:04:56.429046 25950 raft_consensus.cc:2229] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 1 LEADER]: Raft consensus shutting down.
I20240506 20:04:56.429186 25950 raft_consensus.cc:2258] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:56.429442 25950 ts_tablet_manager.cc:1875] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x556ff096f75c std::function<>::operator()() at ??:0
I20240506 20:04:56.430903 25946 tablet_replica.cc:330] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: stopping tablet replica
I20240506 20:04:56.430996 25946 raft_consensus.cc:2229] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 20:04:56.431087 25946 raft_consensus.cc:2258] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:04:56.431195 25950 ts_tablet_manager.cc:1888] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
@ 0x7f5c2563b807 kudu::Thread::SuperviseThread() at ??:0
I20240506 20:04:56.431246 25950 log.cc:1196] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-1/wal/wals/76234ea981fe4ae4a41171ed2586e27e
I20240506 20:04:56.431438 25946 ts_tablet_manager.cc:1875] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:04:56.431504 25950 ts_tablet_manager.cc:1909] T 76234ea981fe4ae4a41171ed2586e27e P c35a1e4b657846ba945aa2602bea72cb: Deleting consensus metadata
I20240506 20:04:56.432052 25840 catalog_manager.cc:4830] TS c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): tablet 76234ea981fe4ae4a41171ed2586e27e (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
I20240506 20:04:56.433056 25946 ts_tablet_manager.cc:1888] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 20:04:56.433104 25946 log.cc:1196] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal/wals/76234ea981fe4ae4a41171ed2586e27e
I20240506 20:04:56.433280 25946 ts_tablet_manager.cc:1909] T 76234ea981fe4ae4a41171ed2586e27e P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting consensus metadata
@ 0x7f5c25e796db start_thread at ??:0
I20240506 20:04:56.433892 25838 catalog_manager.cc:4830] TS b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): tablet 76234ea981fe4ae4a41171ed2586e27e (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
@ 0x7f5c233b171f clone at ??:0
I20240506 20:04:56.449887 25678 raft_consensus.cc:482] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:56.450011 25678 raft_consensus.cc:504] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:56.450579 25678 leader_election.cc:288] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:56.450978 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86be6033bc84646ae3895c2c1242042" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" is_pre_election: true
I20240506 20:04:56.451038 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86be6033bc84646ae3895c2c1242042" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" is_pre_election: true
I20240506 20:04:56.451117 24807 raft_consensus.cc:2454] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 0.
I20240506 20:04:56.451177 24534 raft_consensus.cc:2454] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 0.
I20240506 20:04:56.451524 24604 leader_election.cc:302] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9, c35a1e4b657846ba945aa2602bea72cb; no voters:
I20240506 20:04:56.451716 25678 raft_consensus.cc:2781] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 20:04:56.451756 25678 raft_consensus.cc:482] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 20:04:56.451787 25678 raft_consensus.cc:3037] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:56.452953 25678 raft_consensus.cc:504] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
I20240506 20:04:56.453331 25678 leader_election.cc:288] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 election: Requested vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:56.453606 24807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86be6033bc84646ae3895c2c1242042" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98"
I20240506 20:04:56.453711 24807 raft_consensus.cc:3037] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:56.453781 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86be6033bc84646ae3895c2c1242042" candidate_uuid: "c35a1e4b657846ba945aa2602bea72cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec4b56ff88545a69ab015a9d695d5d9"
I20240506 20:04:56.453887 24534 raft_consensus.cc:3037] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 20:04:56.454347 24807 raft_consensus.cc:2454] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 1.
I20240506 20:04:56.454633 24602 leader_election.cc:302] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb; no voters:
I20240506 20:04:56.454820 24534 raft_consensus.cc:2454] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35a1e4b657846ba945aa2602bea72cb in term 1.
I20240506 20:04:56.455009 25678 raft_consensus.cc:2781] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 1 FOLLOWER]: Leader election won for term 1
I20240506 20:04:56.455192 25678 raft_consensus.cc:686] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [term 1 LEADER]: Becoming Leader. State: Replica: c35a1e4b657846ba945aa2602bea72cb, State: Running, Role: LEADER
I20240506 20:04:56.455353 25678 consensus_queue.cc:237] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } }
W20240506 20:04:56.577023 24860 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.94.125:42097)
W20240506 20:04:56.577401 24860 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.23.94.125:42097)
W20240506 20:04:56.609066 24860 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.23.94.125:42097)
I20240506 20:04:56.858279 25678 consensus_queue.cc:1035] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 20:04:56.862337 25678 consensus_queue.cc:1035] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 20:04:56.914935 25979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:56.915055 25979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:56.915503 25979 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:56.916534 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:56.916638 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:56.916716 25705 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:56.916839 25979 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 20:04:56.987303 25689 master_runner.cc:186] Error getting master registration for 127.23.94.125:42097: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111),
I20240506 20:04:56.987407 25689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:57.596515 25985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:57.596666 25985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:57.597064 25985 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:57.598078 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:04:57.598169 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:57.598227 25705 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:57.598346 25985 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 20:04:58.391121 26001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:58.391263 26001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:58.391680 26001 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:58.392206 25704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240506 20:04:58.392859 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
W20240506 20:04:58.392961 25704 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:04:58.393038 25704 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9; no voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:04:58.393178 26001 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240506 20:04:58.648651 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--webserver_interface=127.23.94.125
--webserver_port=43623
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:04:58.749476 26005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:04:58.749698 26005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:04:58.749779 26005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:04:58.752813 26005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:04:58.752873 26005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:04:58.752899 26005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:04:58.752931 26005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:04:58.756413 26005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.125:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.94.125
--webserver_port=43623
--never_fsync=true
--heap_profile_path=/tmp/kudu.26005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:04:58.757236 26005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:04:58.758282 26005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:04:58.764701 26013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:58.765058 26011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:04:58.765456 26010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:04:58.765928 26005 server_base.cc:1034] running on GCE node
I20240506 20:04:58.766355 26005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:04:58.767174 26005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:04:58.768316 26005 hybrid_clock.cc:648] HybridClock initialized: now 1715025898768304 us; error 27 us; skew 500 ppm
I20240506 20:04:58.774488 26005 webserver.cc:460] Webserver started at http://127.23.94.125:43623/ using document root <none> and password file <none>
I20240506 20:04:58.775017 26005 fs_manager.cc:359] Metadata directory not provided
I20240506 20:04:58.775076 26005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:04:58.778278 26005 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240506 20:04:58.780030 26021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:04:58.780983 26005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:04:58.781093 26005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
uuid: "853e921441fd4ee4b5bb84fdab584029"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:04:58.781486 26005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:04:58.807195 26005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:04:58.807736 26005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:04:58.807878 26005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:04:58.813819 26005 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.125:42097
I20240506 20:04:58.813832 26073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.125:42097 every 8 connection(s)
I20240506 20:04:58.814764 26005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-1/data/info.pb
I20240506 20:04:58.818548 26074 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:04:58.820000 26074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap starting.
I20240506 20:04:58.823817 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 26005
I20240506 20:04:58.825735 26074 log.cc:824] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Log is configured to *not* fsync() on all Append() calls
I20240506 20:04:58.837003 26074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=10 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 20:04:58.837308 26074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Bootstrap complete.
I20240506 20:04:58.839628 26074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:58.840044 26074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853e921441fd4ee4b5bb84fdab584029, State: Initialized, Role: FOLLOWER
I20240506 20:04:58.840449 26074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 8.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:58.840845 26078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:58.840953 26078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:04:58.841395 26074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:04:58.846858 26092 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:58.846917 26092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: loading cluster ID for follower catalog manager: success
I20240506 20:04:58.849303 26092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: acquiring CA information for follower catalog manager: success
I20240506 20:04:58.850837 26092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:04:58.864193 26005 master_runner.cc:186] Error getting master registration for 127.23.94.124:36421: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111),
I20240506 20:04:58.866447 26005 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:04:58.873310 24582 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:58.875345 26038 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:41801; Asking this server to re-register.
I20240506 20:04:58.875905 24582 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:58.877043 26038 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:04:58.878079 24855 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:58.878826 26038 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:50841; Asking this server to re-register.
I20240506 20:04:58.879246 24855 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:58.879925 26038 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:04:58.970039 26103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:58.970165 26103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:58.970594 26103 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:58.972088 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:58.974170 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029" is_pre_election: true
I20240506 20:04:58.974514 26048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 8.
I20240506 20:04:58.974920 25704 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097
I20240506 20:04:58.975101 26103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240506 20:04:58.975170 26103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 853e921441fd4ee4b5bb84fdab584029)
I20240506 20:04:58.975199 26103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 8 FOLLOWER]: Advancing to term 9
I20240506 20:04:58.976042 26103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:58.976363 26103 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 election: Requested vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:04:58.976716 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029"
I20240506 20:04:58.976835 26048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 8 FOLLOWER]: Advancing to term 9
W20240506 20:04:58.977164 25705 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:58.979234 26048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 9.
I20240506 20:04:58.979542 25704 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097
I20240506 20:04:58.979748 26103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 FOLLOWER]: Leader election won for term 9
I20240506 20:04:58.979928 26103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 LEADER]: Becoming Leader. State: Replica: 25192b33b13d4b089dc3fb1d1f6fc2e9, State: Running, Role: LEADER
I20240506 20:04:58.980198 26103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 8.20, Last appended by leader: 20, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:58.981120 26107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25192b33b13d4b089dc3fb1d1f6fc2e9. Latest consensus state: current_term: 9 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:04:58.981209 26107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: LEADER
I20240506 20:04:58.981457 26109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:04:58.982832 26109 catalog_manager.cc:669] Loaded metadata for table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]
I20240506 20:04:58.984061 26109 tablet_loader.cc:96] loaded metadata for tablet d86be6033bc84646ae3895c2c1242042 (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da])
I20240506 20:04:58.984174 26109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:04:58.984591 26109 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:04:58.984691 26109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:04:58.985213 26109 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:04:58.985610 26109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Loaded TSK: 0
I20240506 20:04:58.986210 26109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:04:58.986591 26109 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:04:58.987018 26109 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 11
I20240506 20:04:59.008165 25714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48166:
name: "default.Table_758d3e4f504a489eb12b7bf93d583cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:59.008165 25718 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48166:
name: "default.Table_a1b4d3a0d5ce479584c8f895844cadc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:59.008427 25719 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48166:
name: "default.Table_c568fe36c59c4d93bda72f412845b2f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:59.008608 25711 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48166:
name: "default.Table_7f7588f0e29e4057a66f08687ec28975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:59.008942 25716 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48166:
table { table_name: "default.Table_c66444cb675241d1a23c4348383b208e" } new_table_name: "default.Table_ea33dccc2852419e87b57fc573f6572e" modify_external_catalogs: true
I20240506 20:04:59.008749 25712 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48166:
name: "default.Table_8df5615aacee4940a5402206a921f0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:59.008299 25720 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48166:
name: "default.Table_2ee32586537f466fa07ee5ad6c48dec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:59.008766 25715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48166:
name: "default.Table_6d90713e02f146279b23811c97941079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:59.008862 25717 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48166:
name: "default.Table_6e4d3650620d474ab3107f0f53f5e807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:59.008311 25713 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48166:
name: "default.Table_638fa9e2f7b349a38f86c3d585998fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:04:59.013876 25773 catalog_manager.cc:2706] Deleting table default.table_c568fe36c59c4d93bda72f412845b2f2 [id=f8c2559276c74cc698dc2e82a2d7bf5f] in response to Hive Metastore notification log event 12
W20240506 20:04:59.014124 25773 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_c568fe36c59c4d93bda72f412845b2f2: Not found: the table does not exist: table_id: "f8c2559276c74cc698dc2e82a2d7bf5f" table_name: "default.table_c568fe36c59c4d93bda72f412845b2f2"
F20240506 20:04:59.014189 25773 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 12): 12 CREATE_TABLE default.table_c66444cb675241d1a23c4348383b208e
*** Check failure stack trace: ***
@ 0x7fe66ba4fdcd google::LogMessage::Fail() at ??:0
@ 0x7fe66ba53b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe66ba4f7cc google::LogMessage::Flush() at ??:0
@ 0x7fe66ba4fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fe67097b4f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fe67097a5ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fe670979d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7fe67097efc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55b47ea6475c std::function<>::operator()() at ??:0
@ 0x7fe66ce90807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe66d6ce6db start_thread at ??:0
@ 0x7fe66ac0671f clone at ??:0
W20240506 20:04:59.173274 24861 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.23.94.126:40141)
W20240506 20:04:59.173628 24861 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.94.126:40141)
W20240506 20:04:59.174170 24861 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.23.94.126:40141)
I20240506 20:04:59.271744 24717 heartbeater.cc:347] Connected to a master server at 127.23.94.125:42097
I20240506 20:04:59.272555 26038 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:34229; Asking this server to re-register.
I20240506 20:04:59.272923 24717 heartbeater.cc:464] Registering TS with master...
I20240506 20:04:59.273598 26038 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:04:59.531188 26124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:04:59.531311 26124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:04:59.532395 26124 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:04:59.532912 26023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:59.533586 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:04:59.533689 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:04:59.533733 26023 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:04:59.533913 26124 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240506 20:05:00.249555 26128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:00.249688 26128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:00.250108 26128 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:05:00.251566 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:00.251685 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:05:00.251731 26023 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:05:00.251864 26128 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240506 20:05:01.261915 26134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:01.262043 26134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:01.262400 26134 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:05:01.263621 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:01.263720 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:05:01.263765 26023 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:05:01.263888 26134 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240506 20:05:01.611124 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.126:40141
--webserver_interface=127.23.94.126
--webserver_port=45265
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:05:01.713893 26140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:05:01.714095 26140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:05:01.714147 26140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:05:01.717216 26140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:05:01.717307 26140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:05:01.717347 26140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:05:01.717367 26140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:05:01.720919 26140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.126:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.94.126
--webserver_port=45265
--never_fsync=true
--heap_profile_path=/tmp/kudu.26140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:05:01.721690 26140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:05:01.722678 26140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:05:01.729396 26146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:05:01.729467 26148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:05:01.729832 26145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:05:01.730376 26140 server_base.cc:1034] running on GCE node
I20240506 20:05:01.730880 26140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:05:01.731755 26140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:05:01.732899 26140 hybrid_clock.cc:648] HybridClock initialized: now 1715025901732882 us; error 26 us; skew 500 ppm
I20240506 20:05:01.734606 26140 webserver.cc:460] Webserver started at http://127.23.94.126:45265/ using document root <none> and password file <none>
I20240506 20:05:01.735121 26140 fs_manager.cc:359] Metadata directory not provided
I20240506 20:05:01.735183 26140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:05:01.738287 26140 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240506 20:05:01.740059 26154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:05:01.740988 26140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:05:01.741113 26140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:05:01.741493 26140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:05:01.754601 26140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:05:01.755131 26140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:05:01.755261 26140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:05:01.761221 26140 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.126:40141
I20240506 20:05:01.761229 26206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.126:40141 every 8 connection(s)
I20240506 20:05:01.762183 26140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-0/data/info.pb
I20240506 20:05:01.766000 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 26140
I20240506 20:05:01.766170 26207 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:05:01.767876 26207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Bootstrap starting.
I20240506 20:05:01.774179 26207 log.cc:824] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Log is configured to *not* fsync() on all Append() calls
I20240506 20:05:01.777460 24716 heartbeater.cc:347] Connected to a master server at 127.23.94.126:40141
I20240506 20:05:01.785693 26207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=21 overwritten=0 applied=20 ignored=0} inserts{seen=10 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240506 20:05:01.786032 26207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Bootstrap complete.
I20240506 20:05:01.788457 26207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:01.788969 26207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25192b33b13d4b089dc3fb1d1f6fc2e9, State: Initialized, Role: FOLLOWER
I20240506 20:05:01.789376 26207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 9.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:01.789755 26212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:05:01.789868 26212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:05:01.790370 26207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:05:01.795856 26226 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:05:01.795948 26226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: loading cluster ID for follower catalog manager: success
I20240506 20:05:01.798307 26226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: acquiring CA information for follower catalog manager: success
I20240506 20:05:01.799487 26226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:05:01.807389 26140 master_runner.cc:186] Error getting master registration for 127.23.94.124:36421: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111),
I20240506 20:05:01.810019 26140 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:05:01.875663 24854 heartbeater.cc:347] Connected to a master server at 127.23.94.126:40141
I20240506 20:05:01.876916 26171 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:36821; Asking this server to re-register.
I20240506 20:05:01.877082 24581 heartbeater.cc:347] Connected to a master server at 127.23.94.126:40141
I20240506 20:05:01.877329 24854 heartbeater.cc:464] Registering TS with master...
I20240506 20:05:01.877815 26171 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:41405; Asking this server to re-register.
I20240506 20:05:01.878171 24581 heartbeater.cc:464] Registering TS with master...
I20240506 20:05:01.878417 26170 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:05:01.878948 26171 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:05:02.197013 26212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:02.197224 26212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:02.198339 26212 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:05:02.198696 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 10 candidate_status { last_received { term: 9 index: 21 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029" is_pre_election: true
W20240506 20:05:02.198768 26156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:05:02.198828 26048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 9.
I20240506 20:05:02.199182 26155 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029; no voters:
I20240506 20:05:02.199416 26212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240506 20:05:02.199483 26212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 20:05:02.199522 26212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 9 FOLLOWER]: Advancing to term 10
W20240506 20:05:02.199529 26156 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:05:02.201401 26212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:02.201720 26212 leader_election.cc:288] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 10 election: Requested vote from peers 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:05:02.201987 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" candidate_term: 10 candidate_status { last_received { term: 9 index: 21 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029"
I20240506 20:05:02.202082 26048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 9 FOLLOWER]: Advancing to term 10
W20240506 20:05:02.202596 26156 leader_election.cc:334] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:05:02.202982 26048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25192b33b13d4b089dc3fb1d1f6fc2e9 in term 10.
I20240506 20:05:02.203238 26155 leader_election.cc:302] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097
I20240506 20:05:02.203374 26212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 10 FOLLOWER]: Leader election won for term 10
I20240506 20:05:02.203563 26212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [term 10 LEADER]: Becoming Leader. State: Replica: 25192b33b13d4b089dc3fb1d1f6fc2e9, State: Running, Role: LEADER
I20240506 20:05:02.203851 26212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 9.21, Last appended by leader: 21, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:02.205302 26235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25192b33b13d4b089dc3fb1d1f6fc2e9. Latest consensus state: current_term: 10 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:05:02.205404 26235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: LEADER
I20240506 20:05:02.205672 26237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:05:02.207038 26237 catalog_manager.cc:669] Loaded metadata for table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]
I20240506 20:05:02.208256 26237 tablet_loader.cc:96] loaded metadata for tablet d86be6033bc84646ae3895c2c1242042 (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da])
I20240506 20:05:02.208372 26237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:05:02.208782 26237 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:05:02.208825 26237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:05:02.209352 26237 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:05:02.209758 26237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Loaded TSK: 0
I20240506 20:05:02.210338 26237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:05:02.210731 26237 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:05:02.211089 26237 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 11
I20240506 20:05:02.265353 26171 master_service.cc:946] ReplaceTablet: received request to replace tablet d86be6033bc84646ae3895c2c1242042 from {username='slave'} at 127.0.0.1:43790
I20240506 20:05:02.265730 26167 master_service.cc:946] ReplaceTablet: received request to replace tablet d86be6033bc84646ae3895c2c1242042 from {username='slave'} at 127.0.0.1:43790
I20240506 20:05:02.266184 26163 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_6d90713e02f146279b23811c97941079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:02.266310 26170 master_service.cc:946] ReplaceTablet: received request to replace tablet d86be6033bc84646ae3895c2c1242042 from {username='slave'} at 127.0.0.1:43790
I20240506 20:05:02.266471 26169 master_service.cc:946] ReplaceTablet: received request to replace tablet d86be6033bc84646ae3895c2c1242042 from {username='slave'} at 127.0.0.1:43790
I20240506 20:05:02.266594 26168 master_service.cc:946] ReplaceTablet: received request to replace tablet d86be6033bc84646ae3895c2c1242042 from {username='slave'} at 127.0.0.1:43790
I20240506 20:05:02.266801 26166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_e80158bfa50b427cbd790f160164b525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:02.267074 26165 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_7f7588f0e29e4057a66f08687ec28975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:02.267315 26164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_8df5615aacee4940a5402206a921f0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:02.267549 26162 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_6e4d3650620d474ab3107f0f53f5e807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:02.269564 26048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Refusing update from remote peer 25192b33b13d4b089dc3fb1d1f6fc2e9: Log matching property violated. Preceding OpId in replica: term: 8 index: 20. Preceding OpId from leader: term: 10 index: 23. (index mismatch)
I20240506 20:05:02.270284 26235 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20240506 20:05:02.274284 26225 catalog_manager.cc:2706] Deleting table default.table_c568fe36c59c4d93bda72f412845b2f2 [id=f8c2559276c74cc698dc2e82a2d7bf5f] in response to Hive Metastore notification log event 12
W20240506 20:05:02.274588 26225 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_c568fe36c59c4d93bda72f412845b2f2: Not found: the table does not exist: table_id: "f8c2559276c74cc698dc2e82a2d7bf5f" table_name: "default.table_c568fe36c59c4d93bda72f412845b2f2"
F20240506 20:05:02.274657 26225 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 12): 12 CREATE_TABLE default.table_c66444cb675241d1a23c4348383b208e
*** Check failure stack trace: ***
W20240506 20:05:02.275111 26156 consensus_peers.cc:485] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 -> Peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Couldn't send request to peer 038cb801e9214358bab57093f322c097. Status: Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 20:05:02.276718 26239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25192b33b13d4b089dc3fb1d1f6fc2e9. Latest consensus state: current_term: 10 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:05:02.276819 26239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f25b89d9dcd google::LogMessage::Fail() at ??:0
I20240506 20:05:02.278278 26238 mvcc.cc:205] Tried to move back new op lower bound from 7024746095690305536 to 7024746082225119232. Current Snapshot: MvccSnapshot[applied={T|T < 7024746095690305536}]
I20240506 20:05:02.278426 26235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:05:02.278524 26235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: LEADER
I20240506 20:05:02.278484 26212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:05:02.278573 26212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9 [sys.catalog]: This master's current role is: LEADER
@ 0x7f25b89ddb93 google::LogMessage::SendToLog() at ??:0
I20240506 20:05:02.280570 26167 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:46611; Asking this server to re-register.
@ 0x7f25b89d97cc google::LogMessage::Flush() at ??:0
I20240506 20:05:02.280869 26167 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_a1b4d3a0d5ce479584c8f895844cadc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:02.281095 26239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:05:02.281196 26239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:05:02.281190 26171 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 25192b33b13d4b089dc3fb1d1f6fc2e9: Sending DeleteTablet for 3 replicas of tablet d86be6033bc84646ae3895c2c1242042
W20240506 20:05:02.281970 26171 catalog_manager.cc:5779] Failed to send DeleteReplica request for tablet d86be6033bc84646ae3895c2c1242042: Not found: failed to reset TS proxy: Could not find TS for UUID c35a1e4b657846ba945aa2602bea72cb
@ 0x7f25b89d9a11 google::LogMessage::~LogMessage() at ??:0
I20240506 20:05:02.282287 26171 catalog_manager.cc:6554] ReplaceTablet: tablet d86be6033bc84646ae3895c2c1242042 deleted and replaced by tablet 49874fb075b549f5901a8efc00bb66a6
I20240506 20:05:02.282483 26171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_758d3e4f504a489eb12b7bf93d583cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:02.282801 26170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_638fa9e2f7b349a38f86c3d585998fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20240506 20:05:02.282919 26226 catalog_manager.cc:6107] Error selecting replicas for tablet 49874fb075b549f5901a8efc00bb66a6, reason: Invalid argument: Not enough tablet servers are online for table 'default.table_c66444cb675241d1a23c4348383b208e'. Need at least 3 replicas, but only 2 tablet servers are available
I20240506 20:05:02.282994 24716 heartbeater.cc:464] Registering TS with master...
I20240506 20:05:02.283128 24716 heartbeater.cc:503] Master 127.23.94.126:40141 was elected leader, sending a full tablet report...
I20240506 20:05:02.283571 26169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_2ee32586537f466fa07ee5ad6c48dec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:02.283998 26168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43790:
name: "default.Table_c568fe36c59c4d93bda72f412845b2f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:02.285272 26245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:05:02.285372 26245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f25bd9054f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f25bd9045ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240506 20:05:02.292434 24787 tablet_service.cc:1514] Processing DeleteTablet for tablet d86be6033bc84646ae3895c2c1242042 with delete_type TABLET_DATA_DELETED (replaced by tablet 49874fb075b549f5901a8efc00bb66a6) from {username='slave'} at 127.0.0.1:55868
I20240506 20:05:02.292791 26249 tablet_replica.cc:330] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: stopping tablet replica
I20240506 20:05:02.292923 26249 raft_consensus.cc:2229] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 20:05:02.293040 26249 raft_consensus.cc:2258] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:05:02.293033 24513 tablet_service.cc:1514] Processing DeleteTablet for tablet d86be6033bc84646ae3895c2c1242042 with delete_type TABLET_DATA_DELETED (replaced by tablet 49874fb075b549f5901a8efc00bb66a6) from {username='slave'} at 127.0.0.1:49054
I20240506 20:05:02.293438 26250 tablet_replica.cc:330] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: stopping tablet replica
I20240506 20:05:02.293449 26249 ts_tablet_manager.cc:1875] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240506 20:05:02.293526 26250 raft_consensus.cc:2229] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 20:05:02.293604 26250 raft_consensus.cc:2258] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 20:05:02.293880 26250 ts_tablet_manager.cc:1875] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f25bd903d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20240506 20:05:02.295773 26250 ts_tablet_manager.cc:1888] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 20:05:02.295779 26249 ts_tablet_manager.cc:1888] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240506 20:05:02.295825 26250 log.cc:1196] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-0/wal/wals/d86be6033bc84646ae3895c2c1242042
I20240506 20:05:02.295843 26249 log.cc:1196] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting WAL directory at /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/ts-2/wal/wals/d86be6033bc84646ae3895c2c1242042
I20240506 20:05:02.296012 26250 ts_tablet_manager.cc:1909] T d86be6033bc84646ae3895c2c1242042 P 3ec4b56ff88545a69ab015a9d695d5d9: Deleting consensus metadata
I20240506 20:05:02.296083 26249 ts_tablet_manager.cc:1909] T d86be6033bc84646ae3895c2c1242042 P b0ecd08373ba4b2cb8623b3f4862bb98: Deleting consensus metadata
@ 0x7f25bd908fc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240506 20:05:02.296635 26157 catalog_manager.cc:4830] TS 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): tablet d86be6033bc84646ae3895c2c1242042 (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
I20240506 20:05:02.296726 26155 catalog_manager.cc:4830] TS b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): tablet d86be6033bc84646ae3895c2c1242042 (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]) successfully deleted
@ 0x55a73b64875c std::function<>::operator()() at ??:0
@ 0x7f25b9e1a807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f25ba6586db start_thread at ??:0
@ 0x7f25b7b9071f clone at ??:0
W20240506 20:05:02.448415 24468 connection.cc:535] client connection to 127.23.94.126:40141 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:05:02.448415 24741 connection.cc:535] client connection to 127.23.94.126:40141 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:05:02.448791 24861 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.23.94.126:40141)
W20240506 20:05:02.448937 24861 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.94.126:40141)
W20240506 20:05:02.450083 24716 heartbeater.cc:415] Failed 1 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240506 20:05:02.450181 25641 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 20:05:02.449600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.94.126:40141 (local address 127.0.0.1:43806)
0506 20:05:02.449894 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0506 20:05:02.449908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0506 20:05:02.449988 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 20:05:02.450068 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":196}
W20240506 20:05:02.482196 24861 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.23.94.126:40141)
W20240506 20:05:02.678874 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 1: this message will repeat every 5th retry.
W20240506 20:05:02.731176 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 1: this message will repeat every 5th retry.
I20240506 20:05:02.787618 26266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:05:02.787740 26266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:02.788105 26266 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:05:02.789235 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:02.789355 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:05:02.789397 26023 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:05:02.789515 26266 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 20:05:03.491569 26277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:05:03.491681 26277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:03.491993 26277 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:05:03.493110 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:03.493237 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:05:03.493288 26023 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:05:03.493398 26277 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 20:05:03.867188 26005 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:03.867298 26005 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:05:03.883777 26286 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:03.883894 26286 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:05:03.884287 26286 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:05:03.884984 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:05:03.885118 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:05:03.885177 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:05:03.885306 26286 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:05:04.512347 26289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:05:04.512470 26289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:04.512889 26289 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:05:04.514178 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:04.514283 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 038cb801e9214358bab57093f322c097 (127.23.94.124:36421): Network error: Client connection negotiation failed: client connection to 127.23.94.124:36421: connect: Connection refused (error 111)
I20240506 20:05:04.514330 26023 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 853e921441fd4ee4b5bb84fdab584029; no voters: 038cb801e9214358bab57093f322c097, 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:05:04.514467 26289 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240506 20:05:04.778235 23929 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3taq95/build/debug/bin/kudu
/tmp/dist-test-task3taq95/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--webserver_interface=127.23.94.124
--webserver_port=36977
--builtin_ntp_servers=127.23.94.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240506 20:05:04.878778 26293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 20:05:04.878968 26293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 20:05:04.879009 26293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 20:05:04.882043 26293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 20:05:04.882094 26293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 20:05:04.882112 26293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 20:05:04.882131 26293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 20:05:04.885617 26293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.94.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45719
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.94.124:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.94.124
--webserver_port=36977
--never_fsync=true
--heap_profile_path=/tmp/kudu.26293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type DEBUG
built by None at 06 May 2024 19:43:14 UTC on 1a15035a6830
build id 165
I20240506 20:05:04.886361 26293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 20:05:04.887338 26293 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 20:05:04.894142 26298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:05:04.894223 26299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 20:05:04.894621 26301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 20:05:04.895066 26293 server_base.cc:1034] running on GCE node
I20240506 20:05:04.895483 26293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 20:05:04.896257 26293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 20:05:04.897392 26293 hybrid_clock.cc:648] HybridClock initialized: now 1715025904897380 us; error 22 us; skew 500 ppm
I20240506 20:05:04.915499 26293 webserver.cc:460] Webserver started at http://127.23.94.124:36977/ using document root <none> and password file <none>
I20240506 20:05:04.916028 26293 fs_manager.cc:359] Metadata directory not provided
I20240506 20:05:04.916083 26293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 20:05:04.919296 26293 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:05:04.921051 26307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 20:05:04.921926 26293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 20:05:04.922021 26293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data,/tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
uuid: "038cb801e9214358bab57093f322c097"
format_stamp: "Formatted at 2024-05-06 20:04:33 on dist-test-slave-8c1w"
I20240506 20:05:04.922402 26293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 20:05:04.943197 26293 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 20:05:04.943806 26293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 20:05:04.943946 26293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 20:05:04.950320 26359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.94.124:36421 every 8 connection(s)
I20240506 20:05:04.950321 26293 rpc_server.cc:307] RPC server started. Bound to: 127.23.94.124:36421
I20240506 20:05:04.951301 26293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3taq95/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1715025860818283-23929-0/minicluster-data/master-2/data/info.pb
I20240506 20:05:04.953016 23929 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3taq95/build/debug/bin/kudu as pid 26293
I20240506 20:05:04.955595 26360 sys_catalog.cc:263] Verifying existing consensus state
I20240506 20:05:04.957319 26360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap starting.
I20240506 20:05:04.963111 26360 log.cc:824] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Log is configured to *not* fsync() on all Append() calls
I20240506 20:05:04.972553 26360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=17 ignored=0} inserts{seen=9 ignored=0} mutations{seen=7 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240506 20:05:04.972903 26360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: Bootstrap complete.
I20240506 20:05:04.975232 26360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 7 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:04.975656 26360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038cb801e9214358bab57093f322c097, State: Initialized, Role: FOLLOWER
I20240506 20:05:04.975997 26360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 7.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:04.976320 26364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:05:04.976431 26364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: This master's current role is: FOLLOWER
I20240506 20:05:04.976866 26360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 20:05:04.982636 26378 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:05:04.982744 26378 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: loading cluster ID for follower catalog manager: success
I20240506 20:05:04.980913 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:04.982959 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:05:04.986382 26378 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: acquiring CA information for follower catalog manager: success
I20240506 20:05:04.987779 26378 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240506 20:05:05.042524 24715 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:05:05.044152 26324 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" instance_seqno: 1715025874536027) as {username='slave'} at 127.23.94.66:41763; Asking this server to re-register.
I20240506 20:05:05.044654 24715 heartbeater.cc:464] Registering TS with master...
I20240506 20:05:05.045787 26324 ts_manager.cc:194] Registered new tserver with Master: c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
I20240506 20:05:05.380558 24580 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:05:05.381315 26324 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" instance_seqno: 1715025874365395) as {username='slave'} at 127.23.94.65:33129; Asking this server to re-register.
I20240506 20:05:05.381721 24580 heartbeater.cc:464] Registering TS with master...
I20240506 20:05:05.382580 26324 ts_manager.cc:194] Registered new tserver with Master: 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667)
I20240506 20:05:05.384505 24853 heartbeater.cc:347] Connected to a master server at 127.23.94.124:36421
I20240506 20:05:05.385121 26324 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" instance_seqno: 1715025874707394) as {username='slave'} at 127.23.94.67:55269; Asking this server to re-register.
I20240506 20:05:05.385437 24853 heartbeater.cc:464] Registering TS with master...
I20240506 20:05:05.386009 26324 ts_manager.cc:194] Registered new tserver with Master: b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791)
I20240506 20:05:05.531653 26384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:05.531855 26384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:05.533051 26384 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:05.533594 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:05.534360 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:05:05.536695 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038cb801e9214358bab57093f322c097" candidate_term: 8 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "853e921441fd4ee4b5bb84fdab584029" is_pre_election: true
I20240506 20:05:05.536840 26048 raft_consensus.cc:2354] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 038cb801e9214358bab57093f322c097 for earlier term 8. Current term is 10.
I20240506 20:05:05.537307 26308 leader_election.cc:398] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 038cb801e9214358bab57093f322c097 for earlier term 8. Current term is 10.
I20240506 20:05:05.537353 26308 leader_election.cc:401] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20240506 20:05:05.537510 26384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 7 FOLLOWER]: Advancing to term 10
I20240506 20:05:05.539263 26384 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 10 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 038cb801e9214358bab57093f322c097 for earlier term 8. Current term is 10.
I20240506 20:05:05.939247 26389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:05:05.939373 26389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:05.939776 26389 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
W20240506 20:05:05.940403 26023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240506 20:05:05.941334 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:05:05.943667 26331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 11 candidate_status { last_received { term: 10 index: 24 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097" is_pre_election: true
I20240506 20:05:05.943910 26331 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853e921441fd4ee4b5bb84fdab584029 in term 10.
I20240506 20:05:05.944267 26023 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:05:05.944443 26389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240506 20:05:05.944500 26389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 25192b33b13d4b089dc3fb1d1f6fc2e9)
I20240506 20:05:05.944547 26389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 10 FOLLOWER]: Advancing to term 11
I20240506 20:05:05.945468 26389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:05.945755 26389 leader_election.cc:288] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 election: Requested vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 038cb801e9214358bab57093f322c097 (127.23.94.124:36421)
I20240506 20:05:05.946246 26331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "853e921441fd4ee4b5bb84fdab584029" candidate_term: 11 candidate_status { last_received { term: 10 index: 24 } } ignore_live_leader: false dest_uuid: "038cb801e9214358bab57093f322c097"
I20240506 20:05:05.946365 26331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 10 FOLLOWER]: Advancing to term 11
W20240506 20:05:05.946708 26023 leader_election.cc:334] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:05:05.947260 26331 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853e921441fd4ee4b5bb84fdab584029 in term 11.
I20240506 20:05:05.947513 26023 leader_election.cc:302] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038cb801e9214358bab57093f322c097, 853e921441fd4ee4b5bb84fdab584029; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9
I20240506 20:05:05.947655 26389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 11 FOLLOWER]: Leader election won for term 11
I20240506 20:05:05.947836 26389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [term 11 LEADER]: Becoming Leader. State: Replica: 853e921441fd4ee4b5bb84fdab584029, State: Running, Role: LEADER
I20240506 20:05:05.948109 26389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 10.24, Last appended by leader: 24, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:05.949033 26394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 853e921441fd4ee4b5bb84fdab584029. Latest consensus state: current_term: 11 leader_uuid: "853e921441fd4ee4b5bb84fdab584029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } } }
I20240506 20:05:05.949136 26394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029 [sys.catalog]: This master's current role is: LEADER
I20240506 20:05:05.949417 26396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 20:05:05.950999 26396 catalog_manager.cc:669] Loaded metadata for table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da]
I20240506 20:05:05.952200 26396 tablet_loader.cc:96] loaded metadata for tablet 49874fb075b549f5901a8efc00bb66a6 (table default.table_c66444cb675241d1a23c4348383b208e [id=ee3b4ffa4c254f02b890d1ac3b7261da])
I20240506 20:05:05.952785 26396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 20:05:05.953251 26396 catalog_manager.cc:1259] Loaded cluster ID: 5e07ebc84b4c4150817064ac2fa2374b
I20240506 20:05:05.953301 26396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 20:05:05.953986 26396 catalog_manager.cc:1504] Loading token signing keys...
I20240506 20:05:05.954447 26396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 853e921441fd4ee4b5bb84fdab584029: Loaded TSK: 0
I20240506 20:05:05.955214 26396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 20:05:05.955672 26396 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240506 20:05:05.956104 26396 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 11
I20240506 20:05:05.966849 26037 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53354:
table { table_name: "default.Table_c66444cb675241d1a23c4348383b208e" } new_table_name: "default.Table_ea33dccc2852419e87b57fc573f6572e" modify_external_catalogs: true
I20240506 20:05:05.967034 26038 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53354:
name: "default.Table_c568fe36c59c4d93bda72f412845b2f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:05.967103 26033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53354:
name: "default.Table_a1b4d3a0d5ce479584c8f895844cadc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:05.967371 26034 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53354:
name: "default.Table_758d3e4f504a489eb12b7bf93d583cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:05.967434 26029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53354:
name: "default.Table_7f7588f0e29e4057a66f08687ec28975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:05.967626 26031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53354:
name: "default.Table_6e4d3650620d474ab3107f0f53f5e807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:05.967674 26030 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53354:
name: "default.Table_8df5615aacee4940a5402206a921f0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:05.967746 26036 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53354:
name: "default.Table_2ee32586537f466fa07ee5ad6c48dec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:05.967881 26032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53354:
name: "default.Table_6d90713e02f146279b23811c97941079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:05.967983 26035 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53354:
name: "default.Table_638fa9e2f7b349a38f86c3d585998fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 20:05:05.971756 26091 catalog_manager.cc:2706] Deleting table default.table_c568fe36c59c4d93bda72f412845b2f2 [id=f8c2559276c74cc698dc2e82a2d7bf5f] in response to Hive Metastore notification log event 12
W20240506 20:05:05.971984 26091 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_c568fe36c59c4d93bda72f412845b2f2: Not found: the table does not exist: table_id: "f8c2559276c74cc698dc2e82a2d7bf5f" table_name: "default.table_c568fe36c59c4d93bda72f412845b2f2"
F20240506 20:05:05.972036 26091 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 12): 12 CREATE_TABLE default.table_c66444cb675241d1a23c4348383b208e
*** Check failure stack trace: ***
@ 0x7f0e3fb0ddcd google::LogMessage::Fail() at ??:0
@ 0x7f0e3fb11b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0e3fb0d7cc google::LogMessage::Flush() at ??:0
@ 0x7f0e3fb0da11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f0e44a394f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f0e44a385ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f0e44a37d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7f0e44a3cfc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55823ccc775c std::function<>::operator()() at ??:0
@ 0x7f0e40f4e807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0e4178c6db start_thread at ??:0
@ 0x7f0e3ecc471f clone at ??:0
W20240506 20:05:06.129009 24602 connection.cc:535] client connection to 127.23.94.125:42097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 20:05:06.129963 24860 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.23.94.125:42097)
W20240506 20:05:06.130337 24860 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.94.125:42097)
W20240506 20:05:06.130673 24860 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.23.94.125:42097)
I20240506 20:05:06.903283 26416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:06.903389 26416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:06.903725 26416 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:06.904695 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:06.904811 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:06.904891 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:06.905043 26416 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:05:07.399859 23929 master-stress-test.cc:578] Killing test thread 1/25
W20240506 20:05:07.587425 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 6: this message will repeat every 5th retry.
W20240506 20:05:07.696349 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 6: this message will repeat every 5th retry.
I20240506 20:05:07.983980 26423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:07.984097 26423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:07.984459 26423 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:07.985493 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:07.985579 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:07.985646 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:07.985777 26423 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:05:08.929124 26429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:08.929234 26429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:08.929549 26429 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:08.930707 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:05:08.930845 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:05:08.930967 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:08.931113 26429 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:05:09.551038 26435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:09.551151 26435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:09.551519 26435 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:09.552579 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:09.552685 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:09.552744 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:09.552863 26435 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:05:09.983693 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:09.983803 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:05:12.290619 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 11: this message will repeat every 5th retry.
I20240506 20:05:12.539745 26447 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:12.539862 26447 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:05:12.540320 26447 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:05:12.540952 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:05:12.541059 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:05:12.541118 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:05:12.541234 26447 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:05:12.724416 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 11: this message will repeat every 5th retry.
I20240506 20:05:12.971081 26449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:12.971191 26449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:12.971518 26449 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:12.971956 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240506 20:05:12.972546 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:05:12.972679 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:05:12.972754 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:12.972887 26449 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:05:14.984547 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:14.984725 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:05:16.457768 26462 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:16.457895 26462 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:05:16.458312 26462 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:05:16.458988 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:05:16.459102 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:05:16.459189 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:05:16.459337 26462 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:05:17.193739 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 16: this message will repeat every 5th retry.
I20240506 20:05:17.452536 26465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:17.452697 26465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:17.453109 26465 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:17.454319 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:17.454417 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:17.454493 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:17.454617 26465 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:05:17.688786 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 16: this message will repeat every 5th retry.
I20240506 20:05:18.234863 26471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:18.234982 26471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:18.235342 26471 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:18.235838 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240506 20:05:18.236459 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:18.236536 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:18.236595 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:18.236835 26471 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:05:19.985512 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:19.985613 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:05:20.845139 26484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:20.845270 26484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:20.845656 26484 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:20.846735 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:05:20.846822 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:05:20.846908 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:20.847043 26484 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:05:22.248303 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 21: this message will repeat every 5th retry.
W20240506 20:05:22.769351 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 21: this message will repeat every 5th retry.
I20240506 20:05:24.986359 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:24.986464 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:05:27.095156 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 26: this message will repeat every 5th retry.
W20240506 20:05:27.769874 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 26: this message will repeat every 5th retry.
I20240506 20:05:28.877698 26525 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:28.877846 26525 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:05:28.878374 26525 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:05:28.879082 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:05:28.879184 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:05:28.879269 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:05:28.879415 26525 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:05:29.987324 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:29.987433 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:05:31.930840 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 31: this message will repeat every 5th retry.
W20240506 20:05:32.527995 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 31: this message will repeat every 5th retry.
I20240506 20:05:32.705088 26546 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:32.705211 26546 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:05:32.705641 26546 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:05:32.706352 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:05:32.706481 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:05:32.706564 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:05:32.706692 26546 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:05:34.104780 26550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:34.104920 26550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:34.105340 26550 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:34.105876 26308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240506 20:05:34.106520 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:05:34.106606 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:05:34.106683 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:34.106822 26550 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:05:34.941547 24582 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (57 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:34.989022 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:34.989169 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:05:35.162113 24717 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (124 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:05:35.442544 24855 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (58 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:36.593911 26564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:36.594041 26564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:36.594452 26564 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:36.595467 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:36.595556 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:36.595610 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:36.595733 26564 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:05:37.080838 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 36: this message will repeat every 5th retry.
W20240506 20:05:37.473768 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 36: this message will repeat every 5th retry.
I20240506 20:05:38.071110 26576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:38.071233 26576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:38.071636 26576 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:38.072666 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:38.072757 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:38.072835 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:38.072969 26576 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:05:39.990036 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:39.990164 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:05:42.056907 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 41: this message will repeat every 5th retry.
W20240506 20:05:42.780464 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 41: this message will repeat every 5th retry.
I20240506 20:05:44.990880 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:44.990978 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:05:46.924523 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 46: this message will repeat every 5th retry.
I20240506 20:05:47.597504 26624 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:47.597630 26624 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:05:47.598155 26624 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:05:47.598839 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:05:47.598950 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:05:47.599035 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:05:47.599184 26624 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:05:47.836109 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 46: this message will repeat every 5th retry.
I20240506 20:05:48.521595 26627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:48.521720 26627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:05:48.522127 26627 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:05:48.522598 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240506 20:05:48.523257 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:05:48.523352 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:05:48.523424 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:05:48.523566 26627 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:05:49.991758 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:49.991864 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:05:52.273175 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 51: this message will repeat every 5th retry.
W20240506 20:05:52.665946 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 51: this message will repeat every 5th retry.
I20240506 20:05:54.992667 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:54.992777 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:05:57.503842 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 56: this message will repeat every 5th retry.
W20240506 20:05:57.691517 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 56: this message will repeat every 5th retry.
I20240506 20:05:57.917407 26659 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:05:57.917536 26659 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:05:57.917953 26659 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:05:57.918642 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:05:57.918751 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:05:57.918808 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:05:57.918923 26659 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:05:59.993592 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:05:59.993710 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:06:02.248697 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 61: this message will repeat every 5th retry.
W20240506 20:06:02.750936 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 61: this message will repeat every 5th retry.
I20240506 20:06:04.620381 26680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:04.620520 26680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:04.621065 26680 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:04.621611 26308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:06:04.622148 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:06:04.622229 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:06:04.622285 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:04.622406 26680 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:06:04.994367 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:04.994469 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:06:05.149502 26686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:05.149626 26686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:05.150027 26686 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:05.151053 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:06:05.151137 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:06:05.151214 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:05.151338 26686 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:06:07.188701 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 66: this message will repeat every 5th retry.
W20240506 20:06:07.286971 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 66: this message will repeat every 5th retry.
I20240506 20:06:09.995345 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:09.995462 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:06:10.205719 26708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:10.205852 26708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:10.206297 26708 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:10.206713 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240506 20:06:10.207437 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:06:10.207530 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:06:10.207585 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:10.207715 26708 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:06:12.263090 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 71: this message will repeat every 5th retry.
W20240506 20:06:12.319567 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 71: this message will repeat every 5th retry.
I20240506 20:06:14.996249 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:14.996348 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:06:16.426757 26735 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:16.426940 26735 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:06:16.427537 26735 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:06:16.428140 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:06:16.428239 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:06:16.428319 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:06:16.428444 26735 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:06:17.350061 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 76: this message will repeat every 5th retry.
W20240506 20:06:17.457273 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 76: this message will repeat every 5th retry.
I20240506 20:06:19.997256 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:19.997364 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:06:21.662926 26751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:21.663148 26751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:21.663923 26751 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:21.664430 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:06:21.665068 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:06:21.665158 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:06:21.665233 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:21.665355 26751 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:06:22.124328 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 81: this message will repeat every 5th retry.
W20240506 20:06:22.472932 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 81: this message will repeat every 5th retry.
I20240506 20:06:24.998275 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:24.998382 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:06:27.245882 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 86: this message will repeat every 5th retry.
W20240506 20:06:27.645882 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 86: this message will repeat every 5th retry.
I20240506 20:06:29.999290 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:29.999394 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:06:31.592387 26788 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:31.592501 26788 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:06:31.592947 26788 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:06:31.593477 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:06:31.593643 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:06:31.593721 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:06:31.593848 26788 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:06:32.154363 26791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:32.154480 26791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:32.154844 26791 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:32.155282 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:06:32.155884 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:06:32.155987 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:06:32.156061 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:32.156193 26791 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:06:32.436079 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 91: this message will repeat every 5th retry.
W20240506 20:06:32.549702 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 91: this message will repeat every 5th retry.
I20240506 20:06:33.385010 26797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:33.385120 26797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:33.385506 26797 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:33.386479 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:06:33.386579 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:06:33.386667 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:33.386790 26797 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:06:35.000361 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:35.000484 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:06:35.042858 24582 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (177 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:06:35.543840 24855 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (178 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:06:35.574091 24717 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (377 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:06:36.341058 26814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:36.341194 26814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:36.341574 26814 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:36.342795 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:06:36.342895 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:06:36.342952 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:36.343075 26814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:06:37.450874 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 96: this message will repeat every 5th retry.
W20240506 20:06:37.675241 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 96: this message will repeat every 5th retry.
I20240506 20:06:40.001477 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:40.001605 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:06:40.305344 26837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:40.305474 26837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:40.305905 26837 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:40.306340 26308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240506 20:06:40.307011 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:06:40.307126 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:06:40.307214 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:40.307348 26837 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:06:42.657281 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 101: this message will repeat every 5th retry.
W20240506 20:06:43.094628 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 101: this message will repeat every 5th retry.
I20240506 20:06:45.002722 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:45.002838 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:06:45.684571 26854 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:45.684724 26854 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:06:45.685127 26854 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:06:45.685815 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:06:45.685923 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:06:45.685981 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:06:45.686100 26854 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:06:47.748276 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 106: this message will repeat every 5th retry.
W20240506 20:06:48.143745 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 106: this message will repeat every 5th retry.
I20240506 20:06:49.574235 26863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:49.574354 26863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:49.574724 26863 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:49.575249 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:06:49.575881 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:06:49.575981 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:06:49.576041 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:49.576165 26863 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:06:50.003684 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:50.003793 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:06:50.614228 26874 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:50.614356 26874 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:06:50.614826 26874 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:06:50.615517 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:06:50.615625 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:06:50.615684 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:06:50.615813 26874 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:06:52.581718 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 111: this message will repeat every 5th retry.
W20240506 20:06:53.390847 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 111: this message will repeat every 5th retry.
I20240506 20:06:55.004756 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:06:55.004877 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:06:55.158421 26890 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:55.158557 26890 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:06:55.158972 26890 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:06:55.159618 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:06:55.159796 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:06:55.159876 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:06:55.160003 26890 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:06:57.526719 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 116: this message will repeat every 5th retry.
W20240506 20:06:58.731671 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 116: this message will repeat every 5th retry.
I20240506 20:06:59.233803 26899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:06:59.233937 26899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:06:59.234376 26899 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:06:59.234990 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:06:59.235677 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:06:59.235757 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:06:59.235797 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:06:59.235952 26899 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:07:00.005824 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:00.005951 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:07:02.559883 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 121: this message will repeat every 5th retry.
W20240506 20:07:03.998903 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 121: this message will repeat every 5th retry.
I20240506 20:07:04.086346 26914 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:04.086462 26914 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:07:04.086886 26914 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:07:04.087610 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:07:04.087719 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:07:04.087802 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:07:04.087932 26914 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:07:05.006850 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:05.006969 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:07:05.076819 26313 maintenance_manager.cc:419] P 038cb801e9214358bab57093f322c097: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033361
I20240506 20:07:05.114365 26312 maintenance_manager.cc:643] P 038cb801e9214358bab57093f322c097: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.037s user 0.014s sys 0.000s Metrics: {"bytes_written":7767,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1001,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":403,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":9,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":120}
I20240506 20:07:05.115154 26313 maintenance_manager.cc:419] P 038cb801e9214358bab57093f322c097: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 628 bytes on disk
I20240506 20:07:05.115577 26312 maintenance_manager.cc:643] P 038cb801e9214358bab57093f322c097: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
W20240506 20:07:07.574116 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 126: this message will repeat every 5th retry.
W20240506 20:07:08.775455 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 126: this message will repeat every 5th retry.
I20240506 20:07:09.233515 26930 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:09.233673 26930 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:07:09.234433 26930 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:07:09.235172 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:07:09.235356 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:07:09.235428 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:07:09.235549 26930 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:07:10.007954 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:10.008072 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:07:12.986294 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 131: this message will repeat every 5th retry.
I20240506 20:07:13.015210 26943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:13.015328 26943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:07:13.015698 26943 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:07:13.016196 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:07:13.016810 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:07:13.016898 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:07:13.016979 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:07:13.017114 26943 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:07:13.539680 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 131: this message will repeat every 5th retry.
I20240506 20:07:15.010299 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:15.010430 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:07:16.483929 26987 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:16.484050 26987 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:07:16.484464 26987 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:07:16.485225 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:07:16.485335 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:07:16.485395 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:07:16.485512 26987 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:07:18.312865 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 136: this message will repeat every 5th retry.
W20240506 20:07:18.769901 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 136: this message will repeat every 5th retry.
I20240506 20:07:20.011405 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:20.011529 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:07:23.239019 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 141: this message will repeat every 5th retry.
W20240506 20:07:23.699545 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 141: this message will repeat every 5th retry.
I20240506 20:07:24.868808 27042 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:24.868942 27042 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:07:24.869464 27042 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:07:24.870201 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:07:24.870316 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:07:24.870369 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:07:24.870491 27042 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:07:25.012491 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:25.012611 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:07:27.453253 27060 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:27.453374 27060 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:07:27.453729 27060 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:07:27.454447 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:07:27.454563 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:07:27.454622 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:07:27.454738 27060 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:07:27.741436 27063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:27.741533 27063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:07:27.741854 27063 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:07:27.742288 26308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:07:27.742902 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:07:27.743005 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:07:27.743067 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:07:27.743201 27063 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:07:27.984843 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 146: this message will repeat every 5th retry.
W20240506 20:07:28.510411 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 146: this message will repeat every 5th retry.
I20240506 20:07:30.013564 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:30.013684 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:07:32.986402 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 151: this message will repeat every 5th retry.
W20240506 20:07:33.408203 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 151: this message will repeat every 5th retry.
I20240506 20:07:35.014564 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:35.014698 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:07:35.143493 24582 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (297 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:07:35.644371 24855 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (298 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:07:35.833956 24716 heartbeater.cc:644] Failed to heartbeat to 127.23.94.126:40141 (640 consecutive failures): Network error: Failed to ping master at 127.23.94.126:40141: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:07:36.263445 27127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:36.263624 27127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:07:36.264132 27127 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:07:36.264802 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:07:36.265631 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:07:36.267074 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:07:36.267266 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:07:36.267428 27127 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:07:38.266602 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 156: this message will repeat every 5th retry.
W20240506 20:07:38.443959 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 156: this message will repeat every 5th retry.
I20240506 20:07:40.015682 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:40.015825 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:07:42.655781 27172 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:42.655978 27172 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:07:42.656512 27172 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:07:42.657120 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:07:42.657209 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:07:42.657291 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:07:42.657415 27172 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:07:42.807947 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 161: this message will repeat every 5th retry.
W20240506 20:07:43.262073 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 161: this message will repeat every 5th retry.
I20240506 20:07:45.016863 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:45.017001 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:07:47.851182 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 166: this message will repeat every 5th retry.
W20240506 20:07:48.317340 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 166: this message will repeat every 5th retry.
I20240506 20:07:50.018023 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:50.018157 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:07:50.436844 27231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:50.436986 27231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:07:50.437548 27231 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:07:50.437858 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:07:50.438515 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:07:50.438608 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:07:50.438696 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:07:50.438812 27231 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:07:51.365759 27239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:51.365877 27239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:07:51.366259 27239 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:07:51.367376 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:07:51.367462 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:07:51.367537 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:07:51.367673 27239 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:07:52.362483 27249 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:52.362603 27249 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:07:52.362970 27249 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:07:52.363679 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:07:52.363775 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:07:52.363865 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:07:52.363991 27249 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:07:52.453102 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 171: this message will repeat every 5th retry.
W20240506 20:07:53.308312 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 171: this message will repeat every 5th retry.
I20240506 20:07:54.023252 27261 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:54.023388 27261 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:07:54.023789 27261 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:07:54.024488 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:07:54.024605 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:07:54.024686 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:07:54.024807 27261 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:07:54.270615 27262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:07:54.270763 27262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:07:54.271207 27262 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:07:54.272653 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:07:54.272745 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:07:54.272807 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:07:54.272928 27262 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:07:55.020515 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:07:55.020674 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:07:57.265950 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 176: this message will repeat every 5th retry.
W20240506 20:07:58.169867 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 176: this message will repeat every 5th retry.
I20240506 20:08:00.021687 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:00.021809 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:08:01.921415 27314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:01.921522 27314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:08:01.921871 27314 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:08:01.922263 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240506 20:08:01.923007 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:08:01.923239 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:08:01.923306 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:08:01.923458 27314 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:08:02.355185 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 181: this message will repeat every 5th retry.
W20240506 20:08:02.915482 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 181: this message will repeat every 5th retry.
I20240506 20:08:05.022778 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:05.022915 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:08:07.422241 27351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:07.422451 27351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:08:07.422960 27351 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:08:07.423458 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:08:07.424091 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:08:07.424211 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:08:07.424280 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:08:07.424410 27351 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:08:07.587181 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 186: this message will repeat every 5th retry.
W20240506 20:08:07.840687 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 186: this message will repeat every 5th retry.
I20240506 20:08:08.300002 27360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:08.300132 27360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:08:08.300559 27360 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:08:08.301716 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:08:08.301796 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:08:08.301873 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:08:08.301998 27360 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:08:10.024219 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:10.024343 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:08:10.627254 27377 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:10.627399 27377 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:08:10.627913 27377 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:08:10.628675 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:08:10.628773 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:08:10.628866 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:08:10.629009 27377 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:08:12.178056 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 191: this message will repeat every 5th retry.
W20240506 20:08:12.590926 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 191: this message will repeat every 5th retry.
I20240506 20:08:15.025377 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:15.025507 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:08:15.244736 27406 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:15.244875 27406 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:08:15.245290 27406 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:08:15.245975 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:08:15.246081 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:08:15.246171 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:08:15.246310 27406 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:08:17.011096 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 196: this message will repeat every 5th retry.
W20240506 20:08:17.669967 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 196: this message will repeat every 5th retry.
I20240506 20:08:19.999478 27438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:19.999596 27438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:08:19.999974 27438 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:08:20.000451 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240506 20:08:20.001080 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:08:20.001158 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:08:20.001235 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:08:20.001372 27438 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:08:20.026330 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:20.026443 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:08:22.308738 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 201: this message will repeat every 5th retry.
W20240506 20:08:22.532388 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 201: this message will repeat every 5th retry.
I20240506 20:08:25.027385 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:25.027508 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:08:27.059087 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 206: this message will repeat every 5th retry.
W20240506 20:08:27.601871 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 206: this message will repeat every 5th retry.
I20240506 20:08:30.028512 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:30.028689 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:08:32.137553 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 211: this message will repeat every 5th retry.
I20240506 20:08:32.260452 27518 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:32.260614 27518 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:08:32.261319 27518 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:08:32.262166 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:08:32.262611 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:08:32.262704 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:08:32.262866 27518 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:08:32.833997 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 211: this message will repeat every 5th retry.
I20240506 20:08:34.450969 27529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:34.451128 27529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:08:34.451566 27529 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:08:34.452059 26308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:08:34.452795 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:08:34.452919 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:08:34.452996 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:08:34.453119 27529 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:08:35.029597 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:35.029711 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:08:35.247815 24582 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (417 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:08:35.748494 24855 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (418 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:08:35.988495 24716 heartbeater.cc:644] Failed to heartbeat to 127.23.94.126:40141 (887 consecutive failures): Network error: Failed to ping master at 127.23.94.126:40141: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:08:37.156961 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 216: this message will repeat every 5th retry.
W20240506 20:08:37.833113 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 216: this message will repeat every 5th retry.
I20240506 20:08:40.030673 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:40.030786 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:08:41.903261 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 221: this message will repeat every 5th retry.
W20240506 20:08:42.565837 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 221: this message will repeat every 5th retry.
I20240506 20:08:43.906363 27590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:43.906474 27590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:08:43.906801 27590 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:08:43.907321 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:08:43.908003 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:08:43.908090 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:08:43.908149 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:08:43.908270 27590 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:08:45.031879 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:45.032004 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:08:46.340324 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 226: this message will repeat every 5th retry.
W20240506 20:08:47.837661 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 226: this message will repeat every 5th retry.
I20240506 20:08:50.032966 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:50.033108 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:08:50.472278 27633 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:50.472415 27633 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:08:50.472962 27633 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:08:50.473681 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:08:50.473795 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:08:50.473855 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:08:50.473973 27633 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:08:51.692557 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 231: this message will repeat every 5th retry.
W20240506 20:08:53.010794 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 231: this message will repeat every 5th retry.
I20240506 20:08:55.033996 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:08:55.034114 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:08:56.581907 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 236: this message will repeat every 5th retry.
I20240506 20:08:57.537185 27672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:57.537338 27672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:08:57.537768 27672 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:08:57.538492 26308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:08:57.538492 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:08:57.539259 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:08:57.539341 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:08:57.539397 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:08:57.539533 27672 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:08:58.083410 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 236: this message will repeat every 5th retry.
I20240506 20:08:59.086942 27684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:08:59.087078 27684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:08:59.087493 27684 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:08:59.088583 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:08:59.088706 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:08:59.088793 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:08:59.088930 27684 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:09:00.035108 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:00.035233 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:01.499075 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 241: this message will repeat every 5th retry.
W20240506 20:09:03.165675 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 241: this message will repeat every 5th retry.
I20240506 20:09:05.036113 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:05.036231 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:06.673766 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 246: this message will repeat every 5th retry.
I20240506 20:09:06.872829 27736 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:06.872951 27736 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:09:06.873474 27736 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:09:06.874192 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:09:06.874315 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:09:06.874368 24467 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:09:06.874511 27736 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:09:08.983374 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 246: this message will repeat every 5th retry.
I20240506 20:09:10.037223 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:10.037350 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:11.787174 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 251: this message will repeat every 5th retry.
I20240506 20:09:12.764694 27772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:12.764829 27772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:09:12.765269 27772 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:09:12.765776 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240506 20:09:12.766412 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:09:12.766484 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:09:12.766561 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:09:12.766701 27772 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:09:13.379560 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 251: this message will repeat every 5th retry.
I20240506 20:09:15.038090 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:15.038199 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:09:15.983156 27794 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:15.983275 27794 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:09:15.983646 27794 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:09:15.984325 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:09:15.984431 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:09:15.984511 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:09:15.984656 27794 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:09:16.548643 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 256: this message will repeat every 5th retry.
W20240506 20:09:18.395128 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 256: this message will repeat every 5th retry.
I20240506 20:09:20.004606 27814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:20.004765 27814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:09:20.005142 27814 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:09:20.005595 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:09:20.006168 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:09:20.006249 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:09:20.006326 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:09:20.006464 27814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:09:20.038908 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:20.039014 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:21.641083 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 261: this message will repeat every 5th retry.
W20240506 20:09:23.694083 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 261: this message will repeat every 5th retry.
I20240506 20:09:24.940286 27849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:24.940433 27849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:09:24.940866 27849 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:09:24.941895 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:09:24.941977 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:09:24.942057 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:09:24.942183 27849 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240506 20:09:25.039790 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:25.039894 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:26.884306 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 266: this message will repeat every 5th retry.
I20240506 20:09:28.353389 27871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:28.353547 27871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:09:28.354393 27871 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:09:28.354894 26308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240506 20:09:28.355798 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:09:28.355898 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:09:28.355983 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:09:28.356140 27871 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:09:28.812026 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 266: this message will repeat every 5th retry.
I20240506 20:09:29.082069 27878 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:29.082231 27878 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:09:29.082847 27878 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:09:29.083526 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:09:29.083621 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:09:29.083712 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:09:29.083859 27878 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 20:09:30.040763 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:30.040887 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:32.041024 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 271: this message will repeat every 5th retry.
W20240506 20:09:34.033576 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 271: this message will repeat every 5th retry.
I20240506 20:09:35.041657 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:35.041767 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:35.352326 24582 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (537 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:09:35.853308 24855 heartbeater.cc:644] Failed to heartbeat to 127.23.94.125:42097 (538 consecutive failures): Network error: Failed to ping master at 127.23.94.125:42097: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:09:35.996825 24716 heartbeater.cc:644] Failed to heartbeat to 127.23.94.126:40141 (1129 consecutive failures): Network error: Failed to ping master at 127.23.94.126:40141: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:09:37.035369 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 276: this message will repeat every 5th retry.
W20240506 20:09:38.965701 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 276: this message will repeat every 5th retry.
I20240506 20:09:40.042754 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:40.042887 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:41.532609 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 281: this message will repeat every 5th retry.
W20240506 20:09:44.120054 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 281: this message will repeat every 5th retry.
I20240506 20:09:45.043884 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:45.044011 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240506 20:09:45.835765 27975 raft_consensus.cc:482] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:45.835920 27975 raft_consensus.cc:504] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ecd08373ba4b2cb8623b3f4862bb98" member_type: VOTER last_known_addr { host: "127.23.94.67" port: 44791 } } peers { permanent_uuid: "3ec4b56ff88545a69ab015a9d695d5d9" member_type: VOTER last_known_addr { host: "127.23.94.65" port: 36667 } } peers { permanent_uuid: "c35a1e4b657846ba945aa2602bea72cb" member_type: VOTER last_known_addr { host: "127.23.94.66" port: 35561 } }
I20240506 20:09:45.836434 27975 leader_election.cc:288] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791), c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561)
W20240506 20:09:45.837172 24467 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
W20240506 20:09:45.837284 24469 leader_election.cc:341] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c35a1e4b657846ba945aa2602bea72cb (127.23.94.66:35561): Not found: Tablet not found: 04c1b96ddace45fe8c59d8d38437b0cb
I20240506 20:09:45.837364 24469 leader_election.cc:302] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec4b56ff88545a69ab015a9d695d5d9; no voters: b0ecd08373ba4b2cb8623b3f4862bb98, c35a1e4b657846ba945aa2602bea72cb
I20240506 20:09:45.837498 27975 raft_consensus.cc:2726] T 04c1b96ddace45fe8c59d8d38437b0cb P 3ec4b56ff88545a69ab015a9d695d5d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240506 20:09:46.009989 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 286: this message will repeat every 5th retry.
I20240506 20:09:46.649463 27980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:46.649583 27980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:09:46.650028 27980 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:09:46.650498 26308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:09:46.651134 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:09:46.651218 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:09:46.651296 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:09:46.651428 27980 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:09:48.894591 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 286: this message will repeat every 5th retry.
I20240506 20:09:49.561568 26313 maintenance_manager.cc:419] P 038cb801e9214358bab57093f322c097: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 628 bytes on disk
I20240506 20:09:49.564134 26312 maintenance_manager.cc:643] P 038cb801e9214358bab57093f322c097: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":135,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":180}
I20240506 20:09:50.045220 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:50.045347 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:51.232780 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 291: this message will repeat every 5th retry.
I20240506 20:09:52.228125 28014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:52.228269 28014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:09:52.228700 28014 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:09:52.229328 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:09:52.230082 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
W20240506 20:09:52.230168 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
I20240506 20:09:52.230260 26308 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:09:52.230391 28014 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:09:53.664925 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 291: this message will repeat every 5th retry.
I20240506 20:09:55.046295 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:09:55.046420 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:09:56.442095 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 296: this message will repeat every 5th retry.
I20240506 20:09:58.108533 28047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:09:58.108667 28047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:09:58.109053 28047 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:09:58.109501 26309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240506 20:09:58.110085 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:09:58.110173 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:09:58.110255 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:09:58.110391 28047 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:09:58.534049 24602 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer b0ecd08373ba4b2cb8623b3f4862bb98 (127.23.94.67:44791): Couldn't send request to peer b0ecd08373ba4b2cb8623b3f4862bb98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 296: this message will repeat every 5th retry.
I20240506 20:10:00.047369 26293 master_runner.cc:186] Error getting master registration for 127.23.94.126:40141: Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111),
I20240506 20:10:00.047487 26293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240506 20:10:01.235684 24604 consensus_peers.cc:485] T d86be6033bc84646ae3895c2c1242042 P c35a1e4b657846ba945aa2602bea72cb -> Peer 3ec4b56ff88545a69ab015a9d695d5d9 (127.23.94.65:36667): Couldn't send request to peer 3ec4b56ff88545a69ab015a9d695d5d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86be6033bc84646ae3895c2c1242042. This is attempt 301: this message will repeat every 5th retry.
I20240506 20:10:02.682905 28075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 20:10:02.683029 28075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25192b33b13d4b089dc3fb1d1f6fc2e9" member_type: VOTER last_known_addr { host: "127.23.94.126" port: 40141 } } peers { permanent_uuid: "853e921441fd4ee4b5bb84fdab584029" member_type: VOTER last_known_addr { host: "127.23.94.125" port: 42097 } } peers { permanent_uuid: "038cb801e9214358bab57093f322c097" member_type: VOTER last_known_addr { host: "127.23.94.124" port: 36421 } }
I20240506 20:10:02.683420 28075 leader_election.cc:288] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141), 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097)
W20240506 20:10:02.684460 26308 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 853e921441fd4ee4b5bb84fdab584029 (127.23.94.125:42097): Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:10:02.684559 26309 leader_election.cc:334] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 25192b33b13d4b089dc3fb1d1f6fc2e9 (127.23.94.126:40141): Network error: Client connection negotiation failed: client connection to 127.23.94.126:40141: connect: Connection refused (error 111)
I20240506 20:10:02.684659 26309 leader_election.cc:302] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038cb801e9214358bab57093f322c097; no voters: 25192b33b13d4b089dc3fb1d1f6fc2e9, 853e921441fd4ee4b5bb84fdab584029
I20240506 20:10:02.684789 28075 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 038cb801e9214358bab57093f322c097 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240506 20:10:03.059032 24861 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:10:03.059204 24861 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
W20240506 20:10:03.059247 24861 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
F20240506 20:10:03.069679 24888 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1715026203 (unix time) try "date -d @1715026203" if you are using GNU date ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e800005d79) received by PID 23929 (TID 0x7f41edbc6700) from PID 23929; stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f420bab3980 (unknown) at ??:0
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075388 24874 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e80158bfa50b427cbd790f160164b525 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075388 24874 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e80158bfa50b427cbd790f160164b525 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075410 24867 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8df5615aacee4940a5402206a921f0ce on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075388 24874 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e80158bfa50b427cbd790f160164b525 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075410 24867 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8df5615aacee4940a5402206a921f0ce on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075500 24868 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2ee32586537f466fa07ee5ad6c48dec6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f4209705fb7 gsignal at ??:0
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075388 24874 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e80158bfa50b427cbd790f160164b525 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075410 24867 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8df5615aacee4940a5402206a921f0ce on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075500 24868 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2ee32586537f466fa07ee5ad6c48dec6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076576 24875 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_638fa9e2f7b349a38f86c3d585998fea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075388 24874 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e80158bfa50b427cbd790f160164b525 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075410 24867 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8df5615aacee4940a5402206a921f0ce on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075500 24868 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2ee32586537f466fa07ee5ad6c48dec6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076576 24875 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_638fa9e2f7b349a38f86c3d585998fea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076582 24889 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075388 24874 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e80158bfa50b427cbd790f160164b525 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075410 24867 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8df5615aacee4940a5402206a921f0ce on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075500 24868 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2ee32586537f466fa07ee5ad6c48dec6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076576 24875 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_638fa9e2f7b349a38f86c3d585998fea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076582 24889 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.077750 24891 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075388 24874 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e80158bfa50b427cbd790f160164b525 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075410 24867 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8df5615aacee4940a5402206a921f0ce on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075500 24868 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2ee32586537f466fa07ee5ad6c48dec6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076576 24875 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_638fa9e2f7b349a38f86c3d585998fea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076582 24889 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.077750 24891 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.077805 24873 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_758d3e4f504a489eb12b7bf93d583cb6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f4209707921 abort at ??:0
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075388 24874 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e80158bfa50b427cbd790f160164b525 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075410 24867 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8df5615aacee4940a5402206a921f0ce on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075500 24868 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2ee32586537f466fa07ee5ad6c48dec6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076576 24875 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_638fa9e2f7b349a38f86c3d585998fea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076582 24889 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.077750 24891 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.077805 24873 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_758d3e4f504a489eb12b7bf93d583cb6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.077955 24878 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240506 20:10:03.069713 24870 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6e4d3650620d474ab3107f0f53f5e807 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.070811 24876 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c568fe36c59c4d93bda72f412845b2f2 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.071916 24892 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.073076 24890 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074206 24877 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6d90713e02f146279b23811c97941079 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.074257 24872 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a1b4d3a0d5ce479584c8f895844cadc9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075388 24874 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e80158bfa50b427cbd790f160164b525 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075410 24867 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8df5615aacee4940a5402206a921f0ce on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.075500 24868 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2ee32586537f466fa07ee5ad6c48dec6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076576 24875 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_638fa9e2f7b349a38f86c3d585998fea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.076582 24889 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.077750 24891 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.077805 24873 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_758d3e4f504a489eb12b7bf93d583cb6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.077955 24878 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)F20240506 20:10:03.078969 24869 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7f7588f0e29e4057a66f08687ec28975 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.23.94.126:40141,127.23.94.125:42097,127.23.94.124:36421, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.23.94.125:42097: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f420a572dcd google::LogMessage::Fail() at ??:0
@ 0x7f420a576b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f420a5727cc google::LogMessage::Flush() at ??:0
@ 0x7f420a573f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x561eabc7e5ec kudu::MasterStressTest::ReplaceTabletThread() at ??:0
@ 0x561eabc9076a std::__invoke_impl<>() at ??:0
@ 0x561eabc8e400 std::__invoke<>() at ??:0
@ 0x561eabc95a8f _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x561eabc94b05 std::thread::_Invoker<>::operator()() at ??:0
@ 0x561eabc9392a std::thread::_State_impl<>::_M_run() at ??:0
@ 0x7f4209d8f6df (unknown) at ??:0
@ 0x7f420baa86db start_thread at ??:0
@ 0x7f42097e871f clone at ??:0