Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250109 20:55:04.758431 2433 test_util.cc:274] Using random seed: 826985554
2025-01-09T20:55:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T20:55:04Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskljSzmY/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskljSzmY/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/security.properties
2025-01-09 20:55:11: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskljSzmY/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskljSzmY/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250109 20:55:19.238909 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--webserver_interface=127.2.96.126
--webserver_port=0
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250109 20:55:19.564193 2674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:19.564540 2674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:19.564639 2674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:19.572765 2674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:19.572901 2674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:19.572953 2674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:19.573012 2674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:19.582851 2674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.96.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:19.584571 2674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:19.586320 2674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:19.597311 2679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:19.598330 2674 server_base.cc:1034] running on GCE node
W20250109 20:55:19.599376 2682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:19.599434 2680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:19.600226 2674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:19.601541 2674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:19.602747 2674 hybrid_clock.cc:648] HybridClock initialized: now 1736456119602658 us; error 72 us; skew 500 ppm
I20250109 20:55:19.603073 2674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:19.605403 2674 webserver.cc:458] Webserver started at http://127.2.96.126:37329/ using document root <none> and password file <none>
I20250109 20:55:19.606142 2674 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:19.606264 2674 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:19.606604 2674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:55:19.610183 2674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/instance:
uuid: "b9908c8a5d72498890b9058df83a1b5e"
format_stamp: "Formatted at 2025-01-09 20:55:19 on dist-test-slave-thwl"
I20250109 20:55:19.610980 2674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal/instance:
uuid: "b9908c8a5d72498890b9058df83a1b5e"
format_stamp: "Formatted at 2025-01-09 20:55:19 on dist-test-slave-thwl"
I20250109 20:55:19.615927 2674 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250109 20:55:19.619189 2688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:19.620463 2674 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250109 20:55:19.620605 2674 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
uuid: "b9908c8a5d72498890b9058df83a1b5e"
format_stamp: "Formatted at 2025-01-09 20:55:19 on dist-test-slave-thwl"
I20250109 20:55:19.620775 2674 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:19.639935 2674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:19.640722 2674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:19.640997 2674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:19.658226 2674 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.126:37541
I20250109 20:55:19.658254 2739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.126:37541 every 8 connection(s)
I20250109 20:55:19.659577 2674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
I20250109 20:55:19.663137 2740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:19.665570 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 2674
I20250109 20:55:19.665835 2433 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal/instance
I20250109 20:55:19.668172 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.125:37271
--webserver_interface=127.2.96.125
--webserver_port=0
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:55:19.668989 2740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:55:19.685001 2740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:55:19.686096 2689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:19.687521 2740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.125:37271: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:19.736904 2740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } attempt: 1
W20250109 20:55:19.739077 2740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.125:37271: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:19.807379 2740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } attempt: 2
W20250109 20:55:19.809456 2740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.125:37271: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:19.900823 2740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } attempt: 3
W20250109 20:55:19.902712 2740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.125:37271: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:19.986992 2741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:19.987279 2741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:19.987368 2741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:19.995175 2741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:19.995281 2741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:19.995337 2741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:19.995395 2741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:20.004534 2741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.125:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.96.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:20.006145 2741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:20.007732 2741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:20.017894 2749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:20.018329 2752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:20.019387 2750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:20.020364 2741 server_base.cc:1034] running on GCE node
I20250109 20:55:20.021103 2741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:20.022445 2741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:20.023686 2741 hybrid_clock.cc:648] HybridClock initialized: now 1736456120023625 us; error 34 us; skew 500 ppm
I20250109 20:55:20.024076 2741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:20.026865 2741 webserver.cc:458] Webserver started at http://127.2.96.125:39189/ using document root <none> and password file <none>
I20250109 20:55:20.027944 2741 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:20.028060 2741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:20.028379 2741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:55:20.031215 2741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/instance:
uuid: "d84981b978e3495fb36dc33f09cc825a"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:20.031941 2741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal/instance:
uuid: "d84981b978e3495fb36dc33f09cc825a"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:20.036983 2741 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250109 20:55:20.040216 2758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:20.041656 2741 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250109 20:55:20.041821 2741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
uuid: "d84981b978e3495fb36dc33f09cc825a"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:20.042003 2741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:20.045959 2740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } attempt: 4
W20250109 20:55:20.047850 2740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.125:37271: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:20.067782 2741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:20.068738 2741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:20.069029 2741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:20.086987 2741 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.125:37271
I20250109 20:55:20.087019 2809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.125:37271 every 8 connection(s)
I20250109 20:55:20.088414 2741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
I20250109 20:55:20.092135 2810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:20.094843 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 2741
I20250109 20:55:20.095013 2433 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal/instance
I20250109 20:55:20.096076 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.124:46741
--webserver_interface=127.2.96.124
--webserver_port=0
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:55:20.098111 2810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:55:20.111676 2810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:55:20.120487 2810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:55:20.121448 2762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
W20250109 20:55:20.122889 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:20.172300 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } attempt: 1
W20250109 20:55:20.174278 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:20.242537 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } attempt: 2
W20250109 20:55:20.244452 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:20.335682 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } attempt: 3
W20250109 20:55:20.337697 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:20.347056 2740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } attempt: 5
I20250109 20:55:20.353181 2740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:55:20.354800 2740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:20.406044 2740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } attempt: 1
W20250109 20:55:20.408146 2740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
W20250109 20:55:20.432577 2811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:20.432906 2811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:20.433044 2811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:20.440644 2811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:20.440770 2811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:20.440861 2811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:20.440945 2811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:20.450762 2811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.124:46741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.96.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:20.452461 2811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:20.454144 2811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:20.465742 2820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:20.465971 2819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:20.466542 2822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:20.467469 2811 server_base.cc:1034] running on GCE node
I20250109 20:55:20.468219 2811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:20.469586 2811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:20.470813 2811 hybrid_clock.cc:648] HybridClock initialized: now 1736456120470743 us; error 31 us; skew 500 ppm
I20250109 20:55:20.471117 2811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:20.473466 2811 webserver.cc:458] Webserver started at http://127.2.96.124:36029/ using document root <none> and password file <none>
I20250109 20:55:20.474171 2811 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:20.474292 2811 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:20.474623 2811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:55:20.476373 2740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } attempt: 2
I20250109 20:55:20.477391 2811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/instance:
uuid: "95304c9e18024004b2277a53318ad77e"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
W20250109 20:55:20.478058 2740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:20.478196 2811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal/instance:
uuid: "95304c9e18024004b2277a53318ad77e"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:20.480908 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } attempt: 4
W20250109 20:55:20.482575 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:20.483654 2811 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250109 20:55:20.487479 2828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:20.488765 2811 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250109 20:55:20.488924 2811 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
uuid: "95304c9e18024004b2277a53318ad77e"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:20.489099 2811 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:20.510661 2811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:20.511543 2811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:20.511834 2811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:20.530638 2811 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.124:46741
I20250109 20:55:20.530668 2879 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.124:46741 every 8 connection(s)
I20250109 20:55:20.532058 2811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/info.pb
I20250109 20:55:20.533630 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 2811
I20250109 20:55:20.533828 2433 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal/instance
I20250109 20:55:20.535770 2880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:20.544188 2880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:55:20.556571 2880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:55:20.563731 2880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:55:20.574740 2880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Bootstrap starting.
I20250109 20:55:20.578089 2880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:20.579692 2880 log.cc:826] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:20.582021 2880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: No bootstrap required, opened a new log
I20250109 20:55:20.584285 2740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } attempt: 3
I20250109 20:55:20.590250 2880 raft_consensus.cc:357] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:20.590615 2880 raft_consensus.cc:383] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:20.590741 2880 raft_consensus.cc:738] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95304c9e18024004b2277a53318ad77e, State: Initialized, Role: FOLLOWER
I20250109 20:55:20.591356 2880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:20.592532 2888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.592865 2888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:20.593242 2880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:20.595508 2740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap starting.
I20250109 20:55:20.598948 2740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:20.600114 2740 log.cc:826] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:20.602929 2740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: No bootstrap required, opened a new log
I20250109 20:55:20.614964 2740 raft_consensus.cc:357] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:20.615263 2740 raft_consensus.cc:383] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:20.615413 2740 raft_consensus.cc:738] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Initialized, Role: FOLLOWER
I20250109 20:55:20.616073 2740 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:20.616925 2906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.617295 2906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:20.618122 2740 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:20.608531 2811 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: OK, instance_id { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" instance_seqno: 1736456119653434 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250109 20:55:20.619372 2811 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:55:20.625542 2903 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 20:55:20.625794 2903 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250109 20:55:20.643608 2694 tablet.cc:2367] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:55:20.644902 2674 master_runner.cc:186] Error getting master registration for 127.2.96.125:37271: OK, instance_id { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" instance_seqno: 1736456120081899 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250109 20:55:20.654174 2674 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:55:20.665134 2921 catalog_manager.cc:1559] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 20:55:20.665261 2921 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250109 20:55:20.763834 2834 tablet.cc:2367] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:55:20.781806 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } attempt: 5
I20250109 20:55:20.801010 2810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap starting.
I20250109 20:55:20.804266 2888 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:20.804534 2888 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:20.806078 2810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:20.806249 2888 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
I20250109 20:55:20.807544 2810 log.cc:826] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:20.807412 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9908c8a5d72498890b9058df83a1b5e" is_pre_election: true
I20250109 20:55:20.807940 2715 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95304c9e18024004b2277a53318ad77e in term 0.
I20250109 20:55:20.808674 2832 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e; no voters:
I20250109 20:55:20.809724 2888 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:20.809885 2888 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:20.809994 2888 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:20.813916 2888 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:20.815452 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9908c8a5d72498890b9058df83a1b5e"
I20250109 20:55:20.815671 2715 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:20.817210 2810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: No bootstrap required, opened a new log
I20250109 20:55:20.819358 2715 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95304c9e18024004b2277a53318ad77e in term 1.
I20250109 20:55:20.820025 2832 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e; no voters:
I20250109 20:55:20.827034 2810 raft_consensus.cc:357] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:20.827414 2810 raft_consensus.cc:383] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:20.827553 2810 raft_consensus.cc:738] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84981b978e3495fb36dc33f09cc825a, State: Initialized, Role: FOLLOWER
I20250109 20:55:20.828809 2888 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 1 election: Requested vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
I20250109 20:55:20.828583 2810 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:20.829890 2888 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:20.830605 2888 raft_consensus.cc:695] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 1 LEADER]: Becoming Leader. State: Replica: 95304c9e18024004b2277a53318ad77e, State: Running, Role: LEADER
I20250109 20:55:20.832598 2888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:20.842980 2925 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.843468 2925 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:20.845438 2784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a"
I20250109 20:55:20.845901 2784 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:20.849275 2784 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95304c9e18024004b2277a53318ad77e in term 1.
I20250109 20:55:20.850263 2785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
I20250109 20:55:20.850556 2785 raft_consensus.cc:2371] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 95304c9e18024004b2277a53318ad77e in term 1. Re-sending same reply.
I20250109 20:55:20.850903 2810 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:20.852342 2924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95304c9e18024004b2277a53318ad77e. Latest consensus state: current_term: 1 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.853526 2924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:20.859650 2928 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:20.862007 2928 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:20.883270 2784 raft_consensus.cc:1270] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 1 FOLLOWER]: Refusing update from remote peer 95304c9e18024004b2277a53318ad77e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:55:20.884691 2924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:20.887090 2715 raft_consensus.cc:1270] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 1 FOLLOWER]: Refusing update from remote peer 95304c9e18024004b2277a53318ad77e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:55:20.888273 2888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:20.931262 2906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95304c9e18024004b2277a53318ad77e. Latest consensus state: current_term: 1 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.931569 2906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:20.933887 2930 mvcc.cc:204] Tried to move back new op lower bound from 7112524271125467136 to 7112524270952198144. Current Snapshot: MvccSnapshot[applied={T|T < 7112524271125467136}]
I20250109 20:55:20.941084 2936 mvcc.cc:204] Tried to move back new op lower bound from 7112524271125467136 to 7112524270952198144. Current Snapshot: MvccSnapshot[applied={T|T < 7112524271125467136}]
I20250109 20:55:20.944181 2924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.944458 2924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:20.946475 2928 catalog_manager.cc:1348] Generated new cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:20.946580 2928 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:20.947306 2925 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95304c9e18024004b2277a53318ad77e. Latest consensus state: current_term: 1 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.947587 2925 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:20.946364 2906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.954270 2906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:20.968850 2888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.969265 2888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:20.983026 2925 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:20.983305 2925 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:21.001952 2928 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:55:21.003726 2928 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:21.034312 2928 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Generated new TSK 0
I20250109 20:55:21.035842 2928 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:21.038725 2928 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:21.040370 2928 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:21.046475 2950 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:21.046599 2950 catalog_manager.cc:1553] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: loading cluster ID for follower catalog manager: success
I20250109 20:55:21.052716 2950 catalog_manager.cc:1575] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: acquiring CA information for follower catalog manager: success
I20250109 20:55:21.055408 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.96.65:0
--local_ip_for_outbound_sockets=127.2.96.65
--webserver_interface=127.2.96.65
--webserver_port=0
--tserver_master_addrs=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250109 20:55:21.055766 2950 catalog_manager.cc:1603] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250109 20:55:21.477912 2952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:21.478224 2952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:21.478368 2952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:21.486734 2952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:21.486984 2952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.96.65
I20250109 20:55:21.496752 2952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.96.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.96.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.96.65
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:21.498344 2952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:21.500722 2952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:21.515003 2957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:21.515533 2958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:21.516508 2960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:21.516922 2952 server_base.cc:1034] running on GCE node
I20250109 20:55:21.517560 2952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:21.518586 2952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:21.519824 2952 hybrid_clock.cc:648] HybridClock initialized: now 1736456121519740 us; error 63 us; skew 500 ppm
I20250109 20:55:21.520114 2952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:21.522580 2952 webserver.cc:458] Webserver started at http://127.2.96.65:45611/ using document root <none> and password file <none>
I20250109 20:55:21.523325 2952 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:21.523440 2952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:21.523736 2952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:55:21.526361 2952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/data/instance:
uuid: "0d843776b4dc4cc194def9bd50945c29"
format_stamp: "Formatted at 2025-01-09 20:55:21 on dist-test-slave-thwl"
I20250109 20:55:21.527223 2952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/instance:
uuid: "0d843776b4dc4cc194def9bd50945c29"
format_stamp: "Formatted at 2025-01-09 20:55:21 on dist-test-slave-thwl"
I20250109 20:55:21.533176 2952 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20250109 20:55:21.539814 2967 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95304c9e18024004b2277a53318ad77e)
I20250109 20:55:21.540040 2967 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:21.541854 2967 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:21.542361 2784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
I20250109 20:55:21.542618 2784 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 1.
I20250109 20:55:21.542614 2855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e" is_pre_election: true
I20250109 20:55:21.543119 2689 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters:
I20250109 20:55:21.544696 2967 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 20:55:21.544847 2967 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 95304c9e18024004b2277a53318ad77e)
I20250109 20:55:21.544955 2967 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:55:21.547914 2966 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:21.548698 2967 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:21.549244 2968 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95304c9e18024004b2277a53318ad77e)
I20250109 20:55:21.549293 2967 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 2 election: Requested vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:21.549433 2968 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:21.549779 2784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a"
I20250109 20:55:21.549858 2855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e"
I20250109 20:55:21.550309 2784 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:55:21.550920 2952 fs_manager.cc:730] Time spent opening block manager: real 0.016s user 0.002s sys 0.006s
I20250109 20:55:21.551100 2952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal
uuid: "0d843776b4dc4cc194def9bd50945c29"
format_stamp: "Formatted at 2025-01-09 20:55:21 on dist-test-slave-thwl"
I20250109 20:55:21.551220 2968 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:21.551324 2952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:21.551578 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9908c8a5d72498890b9058df83a1b5e" is_pre_election: true
I20250109 20:55:21.551821 2715 raft_consensus.cc:2388] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d84981b978e3495fb36dc33f09cc825a in current term 2: Already voted for candidate b9908c8a5d72498890b9058df83a1b5e in this term.
I20250109 20:55:21.552850 2855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e" is_pre_election: true
I20250109 20:55:21.553112 2784 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 2.
I20250109 20:55:21.553555 2689 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e
I20250109 20:55:21.553671 2762 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:21.554126 2967 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:55:21.554281 2968 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:55:21.556936 2967 raft_consensus.cc:695] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 2 LEADER]: Becoming Leader. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Running, Role: LEADER
I20250109 20:55:21.557488 2967 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:21.559418 2969 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 2 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:21.559672 2969 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:21.560344 2974 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:21.563300 2974 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:21.564149 2974 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:21.564215 2974 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:21.566684 2974 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:21.567691 2974 catalog_manager.cc:5910] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Loaded TSK: 0
I20250109 20:55:21.568908 2974 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:21.569664 2974 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:21.570375 2974 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:21.582360 2784 raft_consensus.cc:1235] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 2 FOLLOWER]: Rejecting Update request from peer 95304c9e18024004b2277a53318ad77e for earlier term 1. Current term is 2. Ops: []
I20250109 20:55:21.583035 2975 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:55:21.583588 2976 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 1 LEADER]: Stepping down as leader of term 1
I20250109 20:55:21.583701 2976 raft_consensus.cc:738] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 95304c9e18024004b2277a53318ad77e, State: Running, Role: LEADER
I20250109 20:55:21.584024 2976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:21.584451 2976 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:55:21.585691 2952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:21.586541 2952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:21.586876 2952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:21.588121 2952 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:55:21.589574 2952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:55:21.589644 2952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:21.589752 2952 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:55:21.589793 2952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:21.621095 2952 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.65:33615
I20250109 20:55:21.621672 3084 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.65:33615 every 8 connection(s)
I20250109 20:55:21.622757 2952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/data/info.pb
I20250109 20:55:21.628940 2903 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:21.629643 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 2952
I20250109 20:55:21.629889 2433 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/instance
I20250109 20:55:21.637161 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.96.66:0
--local_ip_for_outbound_sockets=127.2.96.66
--webserver_interface=127.2.96.66
--webserver_port=0
--tserver_master_addrs=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250109 20:55:21.661442 3085 heartbeater.cc:346] Connected to a master server at 127.2.96.124:46741
I20250109 20:55:21.661417 3087 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:21.661754 3085 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:21.661790 3087 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:21.663025 3085 heartbeater.cc:510] Master 127.2.96.124:46741 requested a full tablet report, sending...
I20250109 20:55:21.664522 3087 heartbeater.cc:510] Master 127.2.96.125:37271 requested a full tablet report, sending...
I20250109 20:55:21.666654 2845 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:21.669467 2775 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:21.672261 3086 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:21.672375 3086 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:21.672852 3086 heartbeater.cc:510] Master 127.2.96.126:37541 requested a full tablet report, sending...
I20250109 20:55:21.674902 2705 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:21.677054 2705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.96.65:45641
I20250109 20:55:21.967176 2855 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 2 FOLLOWER]: Refusing update from remote peer b9908c8a5d72498890b9058df83a1b5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:55:21.968048 2969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:55:21.969947 2976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 2 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:21.970208 2976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:21.972458 2967 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:21.972802 2967 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:21.973098 2976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:21.973274 2976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:21.974072 2784 raft_consensus.cc:1270] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 2 FOLLOWER]: Refusing update from remote peer b9908c8a5d72498890b9058df83a1b5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250109 20:55:21.974761 2969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250109 20:55:21.976898 3099 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:21.977015 2968 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 2 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:21.977656 3099 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:21.977654 2968 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
W20250109 20:55:22.009109 3094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:22.009399 3094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:22.009502 3094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:22.017585 3094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:22.017870 3094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.96.66
I20250109 20:55:22.027664 3094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.96.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.96.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.96.66
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:22.029443 3094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:22.031126 3094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:22.044211 3104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:22.045248 3107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:22.046109 3105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:22.047500 3094 server_base.cc:1034] running on GCE node
I20250109 20:55:22.048218 3094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:22.049343 3094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:22.050591 3094 hybrid_clock.cc:648] HybridClock initialized: now 1736456122050507 us; error 56 us; skew 500 ppm
I20250109 20:55:22.050936 3094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:22.053378 3094 webserver.cc:458] Webserver started at http://127.2.96.66:42687/ using document root <none> and password file <none>
I20250109 20:55:22.054140 3094 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:22.054271 3094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:22.054601 3094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:55:22.057427 3094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/data/instance:
uuid: "cd7534821da847d0bf53f35562f4b653"
format_stamp: "Formatted at 2025-01-09 20:55:22 on dist-test-slave-thwl"
I20250109 20:55:22.058295 3094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal/instance:
uuid: "cd7534821da847d0bf53f35562f4b653"
format_stamp: "Formatted at 2025-01-09 20:55:22 on dist-test-slave-thwl"
I20250109 20:55:22.063714 3094 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250109 20:55:22.067178 3113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:22.068557 3094 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250109 20:55:22.068758 3094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal
uuid: "cd7534821da847d0bf53f35562f4b653"
format_stamp: "Formatted at 2025-01-09 20:55:22 on dist-test-slave-thwl"
I20250109 20:55:22.068939 3094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:22.087999 3094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:22.089051 3094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:22.089360 3094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:22.090548 3094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:55:22.092475 3094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:55:22.092566 3094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:22.092674 3094 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:55:22.092737 3094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:22.128376 3094 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.66:35185
I20250109 20:55:22.128408 3225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.66:35185 every 8 connection(s)
I20250109 20:55:22.129709 3094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/data/info.pb
I20250109 20:55:22.133890 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 3094
I20250109 20:55:22.134138 2433 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal/instance
I20250109 20:55:22.138906 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.96.67:0
--local_ip_for_outbound_sockets=127.2.96.67
--webserver_interface=127.2.96.67
--webserver_port=0
--tserver_master_addrs=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250109 20:55:22.149995 3228 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:22.150410 3228 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:22.151405 3228 heartbeater.cc:510] Master 127.2.96.125:37271 requested a full tablet report, sending...
I20250109 20:55:22.153049 2775 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:22.153354 3226 heartbeater.cc:346] Connected to a master server at 127.2.96.124:46741
I20250109 20:55:22.153496 3226 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:22.154053 3226 heartbeater.cc:510] Master 127.2.96.124:46741 requested a full tablet report, sending...
I20250109 20:55:22.155421 2845 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:22.157506 3227 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:22.157660 3227 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:22.158818 3227 heartbeater.cc:510] Master 127.2.96.126:37541 requested a full tablet report, sending...
I20250109 20:55:22.159947 2705 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:22.161195 2705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.96.66:55647
I20250109 20:55:22.179827 3086 heartbeater.cc:502] Master 127.2.96.126:37541 was elected leader, sending a full tablet report...
I20250109 20:55:22.383466 2969 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:22.383836 2969 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:22.472605 2976 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:22.472838 2976 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:22.473480 2976 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
I20250109 20:55:22.474028 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9908c8a5d72498890b9058df83a1b5e" is_pre_election: true
I20250109 20:55:22.474115 2784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
I20250109 20:55:22.474843 2832 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:22.475128 2976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250109 20:55:22.490195 3233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:22.490538 3233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:22.490654 3233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:22.498298 3233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:22.498569 3233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.96.67
I20250109 20:55:22.508479 3233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.96.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.96.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.96.67
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:22.510046 3233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:22.511749 3233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:22.524060 3240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:22.525043 3239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:22.525938 3242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:22.526472 3233 server_base.cc:1034] running on GCE node
I20250109 20:55:22.527138 3233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:22.528124 3233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:22.529372 3233 hybrid_clock.cc:648] HybridClock initialized: now 1736456122529298 us; error 55 us; skew 500 ppm
I20250109 20:55:22.529677 3233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:22.532447 3233 webserver.cc:458] Webserver started at http://127.2.96.67:46269/ using document root <none> and password file <none>
I20250109 20:55:22.533412 3233 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:22.533555 3233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:22.533921 3233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:55:22.537449 3233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/data/instance:
uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
format_stamp: "Formatted at 2025-01-09 20:55:22 on dist-test-slave-thwl"
I20250109 20:55:22.538384 3233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal/instance:
uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
format_stamp: "Formatted at 2025-01-09 20:55:22 on dist-test-slave-thwl"
I20250109 20:55:22.545147 3233 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20250109 20:55:22.549679 3248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:22.551124 3233 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250109 20:55:22.551275 3233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal
uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
format_stamp: "Formatted at 2025-01-09 20:55:22 on dist-test-slave-thwl"
I20250109 20:55:22.551455 3233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:22.571444 3233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:22.572256 3233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:22.572527 3233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:22.573982 3233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:55:22.575619 3233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:55:22.575692 3233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:22.575807 3233 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:55:22.575865 3233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:22.608700 3233 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.67:41259
I20250109 20:55:22.608740 3360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.67:41259 every 8 connection(s)
I20250109 20:55:22.609966 3233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/data/info.pb
I20250109 20:55:22.612386 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 3233
I20250109 20:55:22.612563 2433 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal/instance
I20250109 20:55:22.629302 3361 heartbeater.cc:346] Connected to a master server at 127.2.96.124:46741
I20250109 20:55:22.629395 3362 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:22.629668 3361 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:22.629665 3362 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:22.630864 3362 heartbeater.cc:510] Master 127.2.96.126:37541 requested a full tablet report, sending...
I20250109 20:55:22.630887 3361 heartbeater.cc:510] Master 127.2.96.124:46741 requested a full tablet report, sending...
I20250109 20:55:22.632167 3363 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:22.632288 3363 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:22.632511 2705 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:22.632511 2845 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:22.632700 3363 heartbeater.cc:510] Master 127.2.96.125:37271 requested a full tablet report, sending...
I20250109 20:55:22.633533 2705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.96.67:60323
I20250109 20:55:22.633771 2775 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:22.638679 2433 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 20:55:22.663870 3227 heartbeater.cc:502] Master 127.2.96.126:37541 was elected leader, sending a full tablet report...
I20250109 20:55:22.668512 2705 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_f7ca428ec2984ae4992e760dd6c2f8eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:22.669907 2704 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_d2f3663d9ed949609d925f9bb37f7633"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:22.671645 2703 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:22.676939 2702 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_f5f640c97c72444396c24fbb34b9ec18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:22.678656 2701 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_edccb187ef7149b1a1d158edc0924690"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:22.679687 2700 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_94dbbe2d395441859b039d900093b7a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:22.681615 2699 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_320b149b7d80477bb465bb9b3351d0be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:22.681855 2698 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_84db4e769e42410f985d1d5f5cbe527a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:55:22.683151 2705 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f7ca428ec2984ae4992e760dd6c2f8eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:55:22.683614 2697 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_e27d0bbbd6a445f099a3be8e238bf9c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:55:22.687924 2703 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_da5d7a22c7704d4facb0ae576942a0d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:55:22.688458 2704 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d2f3663d9ed949609d925f9bb37f7633 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:55:22.689213 2696 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47678:
name: "default.Table_cbd0b67985f24940af88114907c635eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:55:22.689822 2702 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f5f640c97c72444396c24fbb34b9ec18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:55:22.690799 2701 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_edccb187ef7149b1a1d158edc0924690 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:55:22.708351 2433 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskljSzmY/build/asan/bin/kudu pid 2674
I20250109 20:55:22.710654 2735 generic_service.cc:196] Checking for leaks (request via RPC)
I20250109 20:55:22.864554 2433 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskljSzmY/build/asan/bin/kudu with pid 2674
I20250109 20:55:22.885752 3408 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:22.886016 3408 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:22.886955 3408 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:22.887570 2855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e" is_pre_election: true
W20250109 20:55:22.895126 3371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:22.900261 2762 connection.cc:537] client connection to 127.2.96.126:37541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:22.902832 3374 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:55:22.896566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.96.126:37541 (local address 127.0.0.1:47686)
0109 20:55:22.897243 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0109 20:55:22.897262 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0109 20:55:22.897398 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 20:55:22.901238 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":438}
W20250109 20:55:22.903806 2762 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:22.903980 2762 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:22.904305 3408 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250109 20:55:23.005532 3411 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:23.005734 3411 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:23.012627 3411 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
I20250109 20:55:23.013363 2784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
I20250109 20:55:23.013655 2784 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95304c9e18024004b2277a53318ad77e in term 2.
I20250109 20:55:23.014287 2829 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a; no voters:
W20250109 20:55:23.014361 2832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:23.014770 3411 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250109 20:55:23.014981 3411 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:23.015141 3411 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 2 FOLLOWER]: Advancing to term 3
I20250109 20:55:23.017834 3411 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:23.019447 2784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a"
I20250109 20:55:23.019652 2784 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 2 FOLLOWER]: Advancing to term 3
I20250109 20:55:23.022306 2784 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95304c9e18024004b2277a53318ad77e in term 3.
I20250109 20:55:23.023186 2829 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a; no voters:
I20250109 20:55:23.030910 3414 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 3 FOLLOWER]: Leader election won for term 3
I20250109 20:55:23.031579 3411 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 3 election: Requested vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:23.032941 2832 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:23.033198 3414 raft_consensus.cc:695] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 3 LEADER]: Becoming Leader. State: Replica: 95304c9e18024004b2277a53318ad77e, State: Running, Role: LEADER
I20250109 20:55:23.033650 3414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
W20250109 20:55:23.034312 2832 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:23.037289 3411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95304c9e18024004b2277a53318ad77e. Latest consensus state: current_term: 3 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:23.037737 3411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:23.039527 3417 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:23.042055 3417 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:23.043188 3417 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:23.043272 3417 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:23.044821 3417 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:23.045758 3417 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Loaded TSK: 0
I20250109 20:55:23.046640 3417 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:23.047549 3417 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:23.048408 3417 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:23.059156 2845 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_cbd0b67985f24940af88114907c635eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.058708 2844 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_e27d0bbbd6a445f099a3be8e238bf9c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.061419 2842 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_320b149b7d80477bb465bb9b3351d0be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.061854 2838 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_f5f640c97c72444396c24fbb34b9ec18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.062453 2841 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_d2f3663d9ed949609d925f9bb37f7633"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.062950 2836 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_94dbbe2d395441859b039d900093b7a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.063553 2840 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_f7ca428ec2984ae4992e760dd6c2f8eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.059265 2839 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.062013 2843 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_84db4e769e42410f985d1d5f5cbe527a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.062846 2837 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_edccb187ef7149b1a1d158edc0924690"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:55:23.070111 2845 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cbd0b67985f24940af88114907c635eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:55:23.135861 3362 heartbeater.cc:502] Master 127.2.96.126:37541 was elected leader, sending a full tablet report...
W20250109 20:55:23.144585 3362 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:23.168416 3227 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:23.196072 3086 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:23.221722 2845 catalog_manager.cc:2149] created HMS catalog entry for table [id=206fdcbf4e334b578acb0dc524d09f06]
W20250109 20:55:23.224828 2844 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:55:23.225641 2784 raft_consensus.cc:1270] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 3 FOLLOWER]: Refusing update from remote peer 95304c9e18024004b2277a53318ad77e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
W20250109 20:55:23.226330 2842 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_320b149b7d80477bb465bb9b3351d0be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:55:23.226410 3411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250109 20:55:23.227095 2837 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_edccb187ef7149b1a1d158edc0924690 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:55:23.228116 2832 consensus_peers.cc:487] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e -> Peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Couldn't send request to peer b9908c8a5d72498890b9058df83a1b5e. Status: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250109 20:55:23.228153 2843 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_84db4e769e42410f985d1d5f5cbe527a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:55:23.229187 2839 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_da5d7a22c7704d4facb0ae576942a0d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:55:23.229228 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95304c9e18024004b2277a53318ad77e. Latest consensus state: current_term: 3 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:23.229480 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
W20250109 20:55:23.230119 2840 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f7ca428ec2984ae4992e760dd6c2f8eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:55:23.230746 2836 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_94dbbe2d395441859b039d900093b7a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:55:23.231035 3414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:23.231279 3414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:23.231729 3422 mvcc.cc:204] Tried to move back new op lower bound from 7112524280725426176 to 7112524279949639680. Current Snapshot: MvccSnapshot[applied={T|T < 7112524280725426176}]
W20250109 20:55:23.231889 2841 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d2f3663d9ed949609d925f9bb37f7633 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:55:23.233258 2838 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f5f640c97c72444396c24fbb34b9ec18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:55:23.237283 2845 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56034:
table { table_name: "default.Table_cbd0b67985f24940af88114907c635eb" } new_table_name: "default.Table_61a48f808b814c4aadbc420991d9db06" modify_external_catalogs: true
I20250109 20:55:23.238266 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:23.238521 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:23.280874 3296 tablet_service.cc:1467] Processing CreateTablet for tablet fec5698e37ca4a5b91ed80749b133f4e (DEFAULT_TABLE table=default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.281574 2844 catalog_manager.cc:2149] created HMS catalog entry for table [id=095b9b29244f4bce82fe2b798a9e2a0c]
I20250109 20:55:23.282753 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec5698e37ca4a5b91ed80749b133f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.284085 3020 tablet_service.cc:1467] Processing CreateTablet for tablet fec5698e37ca4a5b91ed80749b133f4e (DEFAULT_TABLE table=default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.285746 3020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec5698e37ca4a5b91ed80749b133f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.290225 2844 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56034:
table { table_name: "default.Table_e27d0bbbd6a445f099a3be8e238bf9c8" } new_table_name: "default.Table_c66ec467f4264b9bbd248d748ba125c0" modify_external_catalogs: true
I20250109 20:55:23.296120 3161 tablet_service.cc:1467] Processing CreateTablet for tablet fec5698e37ca4a5b91ed80749b133f4e (DEFAULT_TABLE table=default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.299026 3161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec5698e37ca4a5b91ed80749b133f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.301967 3160 tablet_service.cc:1467] Processing CreateTablet for tablet 705478e23d324e2a9a059a2227b0b076 (DEFAULT_TABLE table=default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.302620 3160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705478e23d324e2a9a059a2227b0b076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.302827 3295 tablet_service.cc:1467] Processing CreateTablet for tablet 705478e23d324e2a9a059a2227b0b076 (DEFAULT_TABLE table=default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.303354 3295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705478e23d324e2a9a059a2227b0b076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.306895 3019 tablet_service.cc:1467] Processing CreateTablet for tablet 705478e23d324e2a9a059a2227b0b076 (DEFAULT_TABLE table=default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.307464 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705478e23d324e2a9a059a2227b0b076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.325430 3433 tablet_bootstrap.cc:492] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.328471 3431 tablet_bootstrap.cc:492] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:23.331882 3433 tablet_bootstrap.cc:654] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.332505 3431 tablet_bootstrap.cc:654] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.333427 3432 tablet_bootstrap.cc:492] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.334057 3431 log.cc:826] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:23.334048 3433 log.cc:826] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:23.336910 2842 catalog_manager.cc:2149] created HMS catalog entry for table [id=f19741d30cad406b81ee6067dc94b702]
I20250109 20:55:23.338199 3432 tablet_bootstrap.cc:654] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.343055 3432 log.cc:826] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:23.343930 3431 tablet_bootstrap.cc:492] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.344331 3431 ts_tablet_manager.cc:1397] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20250109 20:55:23.354027 3433 tablet_bootstrap.cc:492] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.354362 3433 ts_tablet_manager.cc:1397] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.008s
I20250109 20:55:23.360257 2842 master_service.cc:946] ReplaceTablet: received request to replace tablet fec5698e37ca4a5b91ed80749b133f4e from {username='slave'} at 127.0.0.1:56034
I20250109 20:55:23.362795 3020 tablet_service.cc:1467] Processing CreateTablet for tablet 93e6865a502d40bb986e4df59db8425c (DEFAULT_TABLE table=default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.363031 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 93e6865a502d40bb986e4df59db8425c (DEFAULT_TABLE table=default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.363273 3020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e6865a502d40bb986e4df59db8425c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.363411 3432 tablet_bootstrap.cc:492] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.363487 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e6865a502d40bb986e4df59db8425c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.363711 3432 ts_tablet_manager.cc:1397] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.005s
I20250109 20:55:23.370990 3161 tablet_service.cc:1467] Processing CreateTablet for tablet 93e6865a502d40bb986e4df59db8425c (DEFAULT_TABLE table=default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.371471 3161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e6865a502d40bb986e4df59db8425c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.371541 3431 raft_consensus.cc:357] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.372079 3431 raft_consensus.cc:383] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.372190 3431 raft_consensus.cc:738] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.372921 3431 consensus_queue.cc:260] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.375300 2842 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Sending DeleteTablet for 3 replicas of tablet fec5698e37ca4a5b91ed80749b133f4e
I20250109 20:55:23.376438 3160 tablet_service.cc:1514] Processing DeleteTablet for tablet fec5698e37ca4a5b91ed80749b133f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 0af7427bf3674ed4b00d954366688451) from {username='slave'} at 127.0.0.1:40368
I20250109 20:55:23.376662 3432 raft_consensus.cc:357] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.377169 3432 raft_consensus.cc:383] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.377282 3432 raft_consensus.cc:738] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.377312 2837 catalog_manager.cc:2149] created HMS catalog entry for table [id=f9c3ac24970e4e47af6934218b3e18cc]
I20250109 20:55:23.377269 3296 tablet_service.cc:1514] Processing DeleteTablet for tablet fec5698e37ca4a5b91ed80749b133f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 0af7427bf3674ed4b00d954366688451) from {username='slave'} at 127.0.0.1:49628
I20250109 20:55:23.377429 2842 catalog_manager.cc:6596] ReplaceTablet: tablet fec5698e37ca4a5b91ed80749b133f4e deleted and replaced by tablet 0af7427bf3674ed4b00d954366688451
I20250109 20:55:23.381742 3432 consensus_queue.cc:260] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
W20250109 20:55:23.382941 2832 catalog_manager.cc:4852] TS 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259): delete failed for tablet fec5698e37ca4a5b91ed80749b133f4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fec5698e37ca4a5b91ed80749b133f4e already in progress: creating tablet
I20250109 20:55:23.384552 2837 master_service.cc:946] ReplaceTablet: received request to replace tablet fec5698e37ca4a5b91ed80749b133f4e from {username='slave'} at 127.0.0.1:56034
I20250109 20:55:23.385321 2837 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56034:
table { table_name: "default.Table_320b149b7d80477bb465bb9b3351d0be" } new_table_name: "default.Table_9c2e38f90afc4b9dbf223d20ffa931fb" modify_external_catalogs: true
I20250109 20:55:23.386468 3361 heartbeater.cc:502] Master 127.2.96.124:46741 was elected leader, sending a full tablet report...
I20250109 20:55:23.386598 3085 heartbeater.cc:502] Master 127.2.96.124:46741 was elected leader, sending a full tablet report...
I20250109 20:55:23.387779 3431 ts_tablet_manager.cc:1428] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.043s user 0.015s sys 0.004s
I20250109 20:55:23.392457 3019 tablet_service.cc:1467] Processing CreateTablet for tablet 0af7427bf3674ed4b00d954366688451 (DEFAULT_TABLE table=default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.392915 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af7427bf3674ed4b00d954366688451. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.392808 3159 tablet_service.cc:1467] Processing CreateTablet for tablet 0af7427bf3674ed4b00d954366688451 (DEFAULT_TABLE table=default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.393422 3159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af7427bf3674ed4b00d954366688451. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.395563 3020 tablet_service.cc:1514] Processing DeleteTablet for tablet fec5698e37ca4a5b91ed80749b133f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 0af7427bf3674ed4b00d954366688451) from {username='slave'} at 127.0.0.1:45928
I20250109 20:55:23.399184 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 0af7427bf3674ed4b00d954366688451 (DEFAULT_TABLE table=default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.399679 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af7427bf3674ed4b00d954366688451. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.408210 3431 tablet_bootstrap.cc:492] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:23.408588 3161 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd203495bb5452e9187ee9df32ca046 (DEFAULT_TABLE table=default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.409104 3161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd203495bb5452e9187ee9df32ca046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.410234 3431 tablet_bootstrap.cc:654] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.410671 3019 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd203495bb5452e9187ee9df32ca046 (DEFAULT_TABLE table=default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.411082 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd203495bb5452e9187ee9df32ca046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.411890 3432 ts_tablet_manager.cc:1428] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.048s user 0.014s sys 0.000s
I20250109 20:55:23.420193 3295 tablet_service.cc:1514] Processing DeleteTablet for tablet fec5698e37ca4a5b91ed80749b133f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 0af7427bf3674ed4b00d954366688451) from {username='slave'} at 127.0.0.1:49628
I20250109 20:55:23.420293 2842 master_service.cc:946] ReplaceTablet: received request to replace tablet 705478e23d324e2a9a059a2227b0b076 from {username='slave'} at 127.0.0.1:56034
I20250109 20:55:23.421088 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd203495bb5452e9187ee9df32ca046 (DEFAULT_TABLE table=default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.421020 3433 raft_consensus.cc:357] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.421475 3439 tablet_replica.cc:331] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: stopping tablet replica
I20250109 20:55:23.421605 3433 raft_consensus.cc:383] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.421772 3433 raft_consensus.cc:738] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.422076 3439 raft_consensus.cc:2238] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:23.422524 3439 raft_consensus.cc:2267] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:23.421526 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd203495bb5452e9187ee9df32ca046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.422925 3433 consensus_queue.cc:260] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
W20250109 20:55:23.424304 2830 catalog_manager.cc:4852] TS cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): delete failed for tablet fec5698e37ca4a5b91ed80749b133f4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fec5698e37ca4a5b91ed80749b133f4e already in progress: creating tablet
I20250109 20:55:23.426749 3433 ts_tablet_manager.cc:1428] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.072s user 0.011s sys 0.002s
I20250109 20:55:23.429889 3432 tablet_bootstrap.cc:492] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.427033 3226 heartbeater.cc:502] Master 127.2.96.124:46741 was elected leader, sending a full tablet report...
I20250109 20:55:23.431073 3439 ts_tablet_manager.cc:1905] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:23.432442 3441 tablet_replica.cc:331] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: stopping tablet replica
I20250109 20:55:23.432847 2842 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Sending DeleteTablet for 3 replicas of tablet 705478e23d324e2a9a059a2227b0b076
I20250109 20:55:23.432955 3441 raft_consensus.cc:2238] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:23.433362 3441 raft_consensus.cc:2267] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:23.433952 3433 tablet_bootstrap.cc:492] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.435158 3441 ts_tablet_manager.cc:1905] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:23.435971 3433 tablet_bootstrap.cc:654] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.438519 3432 tablet_bootstrap.cc:654] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.439306 3295 tablet_service.cc:1514] Processing DeleteTablet for tablet 705478e23d324e2a9a059a2227b0b076 with delete_type TABLET_DATA_DELETED (replaced by tablet b685d1fbcd744586b8e0a3024e27a4ff) from {username='slave'} at 127.0.0.1:49628
I20250109 20:55:23.440368 3161 tablet_service.cc:1514] Processing DeleteTablet for tablet 705478e23d324e2a9a059a2227b0b076 with delete_type TABLET_DATA_DELETED (replaced by tablet b685d1fbcd744586b8e0a3024e27a4ff) from {username='slave'} at 127.0.0.1:40368
I20250109 20:55:23.440620 2842 catalog_manager.cc:6596] ReplaceTablet: tablet 705478e23d324e2a9a059a2227b0b076 deleted and replaced by tablet b685d1fbcd744586b8e0a3024e27a4ff
I20250109 20:55:23.440758 3438 tablet_replica.cc:331] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: stopping tablet replica
I20250109 20:55:23.441362 3438 raft_consensus.cc:2238] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:23.441790 3438 raft_consensus.cc:2267] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:23.441962 3019 tablet_service.cc:1514] Processing DeleteTablet for tablet 705478e23d324e2a9a059a2227b0b076 with delete_type TABLET_DATA_DELETED (replaced by tablet b685d1fbcd744586b8e0a3024e27a4ff) from {username='slave'} at 127.0.0.1:45928
I20250109 20:55:23.442940 3432 tablet_bootstrap.cc:492] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.443123 3432 ts_tablet_manager.cc:1397] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250109 20:55:23.443976 3438 ts_tablet_manager.cc:1905] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:23.444171 3432 raft_consensus.cc:357] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.444391 3432 raft_consensus.cc:383] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.444496 3432 raft_consensus.cc:738] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.446496 3431 tablet_bootstrap.cc:492] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.446684 3431 ts_tablet_manager.cc:1397] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20250109 20:55:23.447957 3161 tablet_service.cc:1467] Processing CreateTablet for tablet b685d1fbcd744586b8e0a3024e27a4ff (DEFAULT_TABLE table=default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.448200 3431 raft_consensus.cc:357] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.448398 3161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b685d1fbcd744586b8e0a3024e27a4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.448436 3431 raft_consensus.cc:383] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.448549 3431 raft_consensus.cc:738] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.449102 3296 tablet_service.cc:1467] Processing CreateTablet for tablet b685d1fbcd744586b8e0a3024e27a4ff (DEFAULT_TABLE table=default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.449915 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b685d1fbcd744586b8e0a3024e27a4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.450121 3019 tablet_service.cc:1467] Processing CreateTablet for tablet b685d1fbcd744586b8e0a3024e27a4ff (DEFAULT_TABLE table=default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.449435 3431 consensus_queue.cc:260] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.450525 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b685d1fbcd744586b8e0a3024e27a4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.451824 3431 ts_tablet_manager.cc:1428] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:55:23.452188 3431 tablet_bootstrap.cc:492] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:23.454233 2842 master_service.cc:946] ReplaceTablet: received request to replace tablet fec5698e37ca4a5b91ed80749b133f4e from {username='slave'} at 127.0.0.1:56034
I20250109 20:55:23.456230 3431 tablet_bootstrap.cc:654] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.456525 3432 consensus_queue.cc:260] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.457621 3160 tablet_service.cc:1514] Processing DeleteTablet for tablet fec5698e37ca4a5b91ed80749b133f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 0af7427bf3674ed4b00d954366688451) from {username='slave'} at 127.0.0.1:40368
I20250109 20:55:23.459817 3438 ts_tablet_manager.cc:1918] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:23.460069 3438 log.cc:1198] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal/wals/705478e23d324e2a9a059a2227b0b076
I20250109 20:55:23.460842 3432 ts_tablet_manager.cc:1428] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.018s user 0.000s sys 0.003s
I20250109 20:55:23.460898 3438 ts_tablet_manager.cc:1939] T 705478e23d324e2a9a059a2227b0b076 P cd7534821da847d0bf53f35562f4b653: Deleting consensus metadata
I20250109 20:55:23.462875 2830 catalog_manager.cc:4872] TS cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): tablet 705478e23d324e2a9a059a2227b0b076 (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]) successfully deleted
W20250109 20:55:23.465149 2830 catalog_manager.cc:4852] TS cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): delete failed for tablet fec5698e37ca4a5b91ed80749b133f4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fec5698e37ca4a5b91ed80749b133f4e already in progress: creating tablet
I20250109 20:55:23.460984 3439 ts_tablet_manager.cc:1918] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:23.468152 3439 log.cc:1198] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal/wals/fec5698e37ca4a5b91ed80749b133f4e
I20250109 20:55:23.468683 3439 ts_tablet_manager.cc:1939] T fec5698e37ca4a5b91ed80749b133f4e P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting consensus metadata
I20250109 20:55:23.470417 3439 tablet_replica.cc:331] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: stopping tablet replica
I20250109 20:55:23.470556 2832 catalog_manager.cc:4872] TS 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259): tablet fec5698e37ca4a5b91ed80749b133f4e (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]) successfully deleted
I20250109 20:55:23.470685 3439 raft_consensus.cc:2238] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:23.470836 3439 raft_consensus.cc:2267] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:23.471174 3432 tablet_bootstrap.cc:492] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.471657 2842 master_service.cc:946] ReplaceTablet: received request to replace tablet fec5698e37ca4a5b91ed80749b133f4e from {username='slave'} at 127.0.0.1:56034
I20250109 20:55:23.472682 2842 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56034:
table { table_name: "default.Table_edccb187ef7149b1a1d158edc0924690" } new_table_name: "default.Table_1b53d8836f984435902fbc1c6a9073b1" modify_external_catalogs: true
I20250109 20:55:23.473724 3432 tablet_bootstrap.cc:654] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.475297 3439 ts_tablet_manager.cc:1905] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:23.483592 3431 tablet_bootstrap.cc:492] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.483615 3433 tablet_bootstrap.cc:492] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.483789 3433 ts_tablet_manager.cc:1397] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.011s
I20250109 20:55:23.483809 3431 ts_tablet_manager.cc:1397] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250109 20:55:23.484652 3431 raft_consensus.cc:357] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.484675 3433 raft_consensus.cc:357] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.484894 3433 raft_consensus.cc:383] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.484900 3431 raft_consensus.cc:383] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.484982 3431 raft_consensus.cc:738] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.485278 3431 consensus_queue.cc:260] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.485852 3431 ts_tablet_manager.cc:1428] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:55:23.486135 3433 raft_consensus.cc:738] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.486443 2843 catalog_manager.cc:2149] created HMS catalog entry for table [id=c6228f0aa09b45e2b416a88a82683869]
I20250109 20:55:23.486486 3433 consensus_queue.cc:260] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.490024 3439 ts_tablet_manager.cc:1918] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:23.490186 3439 log.cc:1198] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal/wals/705478e23d324e2a9a059a2227b0b076
I20250109 20:55:23.490589 3439 ts_tablet_manager.cc:1939] T 705478e23d324e2a9a059a2227b0b076 P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting consensus metadata
I20250109 20:55:23.491557 3441 ts_tablet_manager.cc:1918] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:23.491765 3441 log.cc:1198] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/wals/fec5698e37ca4a5b91ed80749b133f4e
I20250109 20:55:23.492153 3433 ts_tablet_manager.cc:1428] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250109 20:55:23.492278 3441 ts_tablet_manager.cc:1939] T fec5698e37ca4a5b91ed80749b133f4e P 0d843776b4dc4cc194def9bd50945c29: Deleting consensus metadata
I20250109 20:55:23.495292 3441 tablet_replica.cc:331] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: stopping tablet replica
I20250109 20:55:23.495492 3441 raft_consensus.cc:2238] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:23.495652 3441 raft_consensus.cc:2267] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:23.496152 3161 tablet_service.cc:1514] Processing DeleteTablet for tablet fec5698e37ca4a5b91ed80749b133f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 0af7427bf3674ed4b00d954366688451) from {username='slave'} at 127.0.0.1:40368
I20250109 20:55:23.496431 2831 catalog_manager.cc:4872] TS 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): tablet fec5698e37ca4a5b91ed80749b133f4e (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]) successfully deleted
I20250109 20:55:23.496480 3438 tablet_replica.cc:331] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: stopping tablet replica
I20250109 20:55:23.496726 3438 raft_consensus.cc:2238] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:23.496944 3438 raft_consensus.cc:2267] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:23.497624 2843 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_5dcc9f9689084615b4c47a34fdd1a329"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.498397 3438 ts_tablet_manager.cc:1905] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:23.501116 3019 tablet_service.cc:1467] Processing CreateTablet for tablet d8d7677ecbb9460192ca4323d38b03ad (DEFAULT_TABLE table=default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.501531 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d7677ecbb9460192ca4323d38b03ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.507382 3161 tablet_service.cc:1467] Processing CreateTablet for tablet d8d7677ecbb9460192ca4323d38b03ad (DEFAULT_TABLE table=default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.507947 3161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d7677ecbb9460192ca4323d38b03ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.508886 3296 tablet_service.cc:1467] Processing CreateTablet for tablet d8d7677ecbb9460192ca4323d38b03ad (DEFAULT_TABLE table=default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.511749 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d7677ecbb9460192ca4323d38b03ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.518940 2832 catalog_manager.cc:4872] TS 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259): tablet 705478e23d324e2a9a059a2227b0b076 (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]) successfully deleted
I20250109 20:55:23.519093 3431 tablet_bootstrap.cc:492] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:23.520921 3431 tablet_bootstrap.cc:654] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.524322 3438 ts_tablet_manager.cc:1918] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:23.524531 3438 log.cc:1198] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal/wals/fec5698e37ca4a5b91ed80749b133f4e
I20250109 20:55:23.525030 3438 ts_tablet_manager.cc:1939] T fec5698e37ca4a5b91ed80749b133f4e P cd7534821da847d0bf53f35562f4b653: Deleting consensus metadata
I20250109 20:55:23.525794 2830 catalog_manager.cc:4872] TS cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): tablet fec5698e37ca4a5b91ed80749b133f4e (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]) successfully deleted
I20250109 20:55:23.527333 3433 tablet_bootstrap.cc:492] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.529953 3433 tablet_bootstrap.cc:654] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.532258 3441 ts_tablet_manager.cc:1905] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:23.533190 2839 catalog_manager.cc:2149] created HMS catalog entry for table [id=6ef574c52a36423db5b9b06b9909f0ec]
I20250109 20:55:23.539718 3432 tablet_bootstrap.cc:492] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.539927 3432 ts_tablet_manager.cc:1397] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.069s user 0.016s sys 0.031s
I20250109 20:55:23.540786 3432 raft_consensus.cc:357] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.541020 3432 raft_consensus.cc:383] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.541095 3432 raft_consensus.cc:738] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.541424 3432 consensus_queue.cc:260] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.541777 2839 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_e992250ce97b4de3ac9cf07ff61775a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:23.541985 3441 ts_tablet_manager.cc:1918] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:23.542143 3441 log.cc:1198] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/wals/705478e23d324e2a9a059a2227b0b076
I20250109 20:55:23.542598 3441 ts_tablet_manager.cc:1939] T 705478e23d324e2a9a059a2227b0b076 P 0d843776b4dc4cc194def9bd50945c29: Deleting consensus metadata
I20250109 20:55:23.543720 3161 tablet_service.cc:1514] Processing DeleteTablet for tablet fec5698e37ca4a5b91ed80749b133f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 0af7427bf3674ed4b00d954366688451) from {username='slave'} at 127.0.0.1:40368
I20250109 20:55:23.544462 3432 ts_tablet_manager.cc:1428] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:55:23.544767 3432 tablet_bootstrap.cc:492] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
W20250109 20:55:23.545622 2830 catalog_manager.cc:4839] TS cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): delete failed for tablet fec5698e37ca4a5b91ed80749b133f4e because the tablet was not found. No further retry: Not found: Tablet not found: fec5698e37ca4a5b91ed80749b133f4e
I20250109 20:55:23.545787 3431 tablet_bootstrap.cc:492] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.545938 3431 ts_tablet_manager.cc:1397] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250109 20:55:23.546702 3432 tablet_bootstrap.cc:654] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.546779 3431 raft_consensus.cc:357] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.546988 3431 raft_consensus.cc:383] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.547077 3431 raft_consensus.cc:738] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.547375 3431 consensus_queue.cc:260] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.547979 2831 catalog_manager.cc:4872] TS 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): tablet 705478e23d324e2a9a059a2227b0b076 (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]) successfully deleted
I20250109 20:55:23.549858 3431 ts_tablet_manager.cc:1428] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:55:23.553680 3431 tablet_bootstrap.cc:492] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:23.554224 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae383e807bc4040b00d6c1afcdfdcfe (DEFAULT_TABLE table=default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.554659 3161 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae383e807bc4040b00d6c1afcdfdcfe (DEFAULT_TABLE table=default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.554726 3019 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae383e807bc4040b00d6c1afcdfdcfe (DEFAULT_TABLE table=default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.555140 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae383e807bc4040b00d6c1afcdfdcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.555171 3161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae383e807bc4040b00d6c1afcdfdcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.556087 3432 tablet_bootstrap.cc:492] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.556259 3432 ts_tablet_manager.cc:1397] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250109 20:55:23.557195 3432 raft_consensus.cc:357] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.557492 3432 raft_consensus.cc:383] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.562207 3432 raft_consensus.cc:738] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.562630 3432 consensus_queue.cc:260] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.564530 3432 ts_tablet_manager.cc:1428] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250109 20:55:23.564890 3432 tablet_bootstrap.cc:492] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.566619 3432 tablet_bootstrap.cc:654] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.567239 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae383e807bc4040b00d6c1afcdfdcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.568519 3432 tablet_bootstrap.cc:492] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.568766 3432 ts_tablet_manager.cc:1397] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250109 20:55:23.568832 3433 tablet_bootstrap.cc:492] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.569020 3433 ts_tablet_manager.cc:1397] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.007s
I20250109 20:55:23.569685 3432 raft_consensus.cc:357] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } }
I20250109 20:55:23.569763 3433 raft_consensus.cc:357] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.569990 3433 raft_consensus.cc:383] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.570063 3433 raft_consensus.cc:738] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.570094 3432 raft_consensus.cc:383] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.570371 3432 raft_consensus.cc:738] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.570374 3433 consensus_queue.cc:260] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.570801 3432 consensus_queue.cc:260] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } }
I20250109 20:55:23.571468 3432 ts_tablet_manager.cc:1428] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:55:23.571882 3432 tablet_bootstrap.cc:492] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.573256 3433 ts_tablet_manager.cc:1428] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250109 20:55:23.574496 3433 tablet_bootstrap.cc:492] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.574849 3432 tablet_bootstrap.cc:654] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.576238 3433 tablet_bootstrap.cc:654] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.576790 3431 tablet_bootstrap.cc:654] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.582959 3431 tablet_bootstrap.cc:492] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.583160 3431 ts_tablet_manager.cc:1397] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250109 20:55:23.583961 3431 raft_consensus.cc:357] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } }
I20250109 20:55:23.584196 3431 raft_consensus.cc:383] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.584300 3431 raft_consensus.cc:738] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.584614 3431 consensus_queue.cc:260] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } }
I20250109 20:55:23.588994 3431 ts_tablet_manager.cc:1428] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250109 20:55:23.592120 3431 tablet_bootstrap.cc:492] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:23.594345 3431 tablet_bootstrap.cc:654] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.594478 3433 tablet_bootstrap.cc:492] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.594640 3433 ts_tablet_manager.cc:1397] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20250109 20:55:23.595800 3433 raft_consensus.cc:357] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.596097 3433 raft_consensus.cc:383] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.596210 3433 raft_consensus.cc:738] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.596624 3432 tablet_bootstrap.cc:492] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.596524 3433 consensus_queue.cc:260] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.596938 3432 ts_tablet_manager.cc:1397] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.010s
I20250109 20:55:23.597857 3432 raft_consensus.cc:357] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.598183 3432 raft_consensus.cc:383] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.598237 3433 ts_tablet_manager.cc:1428] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:55:23.598392 3432 raft_consensus.cc:738] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.598706 3432 consensus_queue.cc:260] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.599334 3432 ts_tablet_manager.cc:1428] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:55:23.599457 3433 tablet_bootstrap.cc:492] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.599720 3432 tablet_bootstrap.cc:492] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.601240 3433 tablet_bootstrap.cc:654] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.601795 3432 tablet_bootstrap.cc:654] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.604085 3440 raft_consensus.cc:491] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:23.604583 3431 tablet_bootstrap.cc:492] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.604391 3440 raft_consensus.cc:513] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } }
I20250109 20:55:23.604730 3431 ts_tablet_manager.cc:1397] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20250109 20:55:23.605492 3431 raft_consensus.cc:357] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.605698 3431 raft_consensus.cc:383] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.605803 3431 raft_consensus.cc:738] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.606065 3431 consensus_queue.cc:260] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.607021 3440 leader_election.cc:290] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.619194 3431 ts_tablet_manager.cc:1428] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
W20250109 20:55:23.619982 3364 tablet.cc:2367] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:55:23.621609 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd203495bb5452e9187ee9df32ca046" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:23.626019 3431 tablet_bootstrap.cc:492] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
W20250109 20:55:23.626507 3088 tablet.cc:2367] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250109 20:55:23.627631 2971 leader_election.cc:343] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:55:23.628024 2836 catalog_manager.cc:2149] created HMS catalog entry for table [id=83f5ad9633e74385904e3e6cfa452ecc]
I20250109 20:55:23.629702 3431 tablet_bootstrap.cc:654] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.637188 2836 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56034:
table { table_name: "default.Table_84db4e769e42410f985d1d5f5cbe527a" } new_table_name: "default.Table_a2481e08ab1c4f2193462e658e9dbecc" modify_external_catalogs: true
I20250109 20:55:23.638986 3432 tablet_bootstrap.cc:492] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.639144 3431 tablet_bootstrap.cc:492] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.639195 3432 ts_tablet_manager.cc:1397] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.040s user 0.020s sys 0.018s
I20250109 20:55:23.640127 3432 raft_consensus.cc:357] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.640367 3432 raft_consensus.cc:383] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.640465 3432 raft_consensus.cc:738] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.640686 3431 ts_tablet_manager.cc:1397] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
W20250109 20:55:23.641096 3229 tablet.cc:2367] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:55:23.642308 3432 consensus_queue.cc:260] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.642109 3431 raft_consensus.cc:357] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.642825 3431 raft_consensus.cc:383] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.643186 3431 raft_consensus.cc:738] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.643543 3431 consensus_queue.cc:260] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.643672 3440 raft_consensus.cc:491] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:23.643915 3440 raft_consensus.cc:513] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.646677 3431 ts_tablet_manager.cc:1428] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250109 20:55:23.646963 3440 leader_election.cc:290] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.648392 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d7677ecbb9460192ca4323d38b03ad" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:23.649987 3431 tablet_bootstrap.cc:492] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
W20250109 20:55:23.650936 2971 leader_election.cc:343] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:55:23.652360 3019 tablet_service.cc:1467] Processing CreateTablet for tablet 8a47d3e2c8ea4690bbdbe87936af9415 (DEFAULT_TABLE table=default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.652647 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 8a47d3e2c8ea4690bbdbe87936af9415 (DEFAULT_TABLE table=default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.652769 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a47d3e2c8ea4690bbdbe87936af9415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.652992 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a47d3e2c8ea4690bbdbe87936af9415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.653074 3440 raft_consensus.cc:491] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:23.653306 3440 raft_consensus.cc:513] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.654776 3440 leader_election.cc:290] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.654875 3315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d7677ecbb9460192ca4323d38b03ad" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:23.655406 3315 raft_consensus.cc:2463] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:23.655522 3314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b685d1fbcd744586b8e0a3024e27a4ff" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:23.655790 3314 raft_consensus.cc:2463] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:23.655992 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b685d1fbcd744586b8e0a3024e27a4ff" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:23.656077 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd203495bb5452e9187ee9df32ca046" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:23.656347 3316 raft_consensus.cc:2463] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
W20250109 20:55:23.656674 2971 leader_election.cc:343] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:55:23.657943 3161 tablet_service.cc:1467] Processing CreateTablet for tablet 8a47d3e2c8ea4690bbdbe87936af9415 (DEFAULT_TABLE table=default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.658375 3161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a47d3e2c8ea4690bbdbe87936af9415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.661650 2973 leader_election.cc:304] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters: cd7534821da847d0bf53f35562f4b653
I20250109 20:55:23.662532 2973 leader_election.cc:304] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters: cd7534821da847d0bf53f35562f4b653
I20250109 20:55:23.662999 2973 leader_election.cc:304] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters: cd7534821da847d0bf53f35562f4b653
I20250109 20:55:23.663208 3440 raft_consensus.cc:2798] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:23.663653 3440 raft_consensus.cc:491] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:23.663811 3440 raft_consensus.cc:3054] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.666719 3440 raft_consensus.cc:513] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.667240 3433 tablet_bootstrap.cc:492] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.667425 3433 ts_tablet_manager.cc:1397] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.010s
I20250109 20:55:23.668303 3431 tablet_bootstrap.cc:654] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.668215 3433 raft_consensus.cc:357] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } }
I20250109 20:55:23.668455 3433 raft_consensus.cc:383] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.668521 3433 raft_consensus.cc:738] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.668897 3440 leader_election.cc:290] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Requested vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.668812 3433 consensus_queue.cc:260] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } }
I20250109 20:55:23.669464 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d7677ecbb9460192ca4323d38b03ad" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653"
I20250109 20:55:23.671048 3433 ts_tablet_manager.cc:1428] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:55:23.671375 3440 raft_consensus.cc:2798] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:23.671802 3440 raft_consensus.cc:491] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:23.672089 3440 raft_consensus.cc:3054] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.672307 3433 tablet_bootstrap.cc:492] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
W20250109 20:55:23.670055 2971 leader_election.cc:343] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:55:23.675427 3433 tablet_bootstrap.cc:654] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.681947 3440 raft_consensus.cc:513] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.683941 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d7677ecbb9460192ca4323d38b03ad" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
I20250109 20:55:23.684163 3316 raft_consensus.cc:3054] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.687917 3316 raft_consensus.cc:2463] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:23.688108 3433 tablet_bootstrap.cc:492] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.688268 3433 ts_tablet_manager.cc:1397] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20250109 20:55:23.689141 3433 raft_consensus.cc:357] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.689342 3433 raft_consensus.cc:383] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.689440 3433 raft_consensus.cc:738] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.689803 3431 tablet_bootstrap.cc:492] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.689713 3433 consensus_queue.cc:260] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.690048 3431 ts_tablet_manager.cc:1397] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20250109 20:55:23.690896 3431 raft_consensus.cc:357] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.691937 3433 ts_tablet_manager.cc:1428] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:55:23.682833 3461 raft_consensus.cc:2798] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:23.692783 3461 raft_consensus.cc:491] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:23.693014 3461 raft_consensus.cc:3054] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.688683 2973 leader_election.cc:304] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters: cd7534821da847d0bf53f35562f4b653
I20250109 20:55:23.693109 3433 tablet_bootstrap.cc:492] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.693742 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b685d1fbcd744586b8e0a3024e27a4ff" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653"
I20250109 20:55:23.693958 3181 raft_consensus.cc:3054] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.695240 3433 tablet_bootstrap.cc:654] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.697207 3181 raft_consensus.cc:2463] T b685d1fbcd744586b8e0a3024e27a4ff P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:23.699602 2971 leader_election.cc:304] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:23.700769 3440 leader_election.cc:290] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Requested vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.700520 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b685d1fbcd744586b8e0a3024e27a4ff" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
I20250109 20:55:23.701049 3316 raft_consensus.cc:3054] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.704443 3316 raft_consensus.cc:2463] T b685d1fbcd744586b8e0a3024e27a4ff P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:23.710475 3431 raft_consensus.cc:383] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.710804 3440 raft_consensus.cc:2798] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:23.710963 3431 raft_consensus.cc:738] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.711298 3431 consensus_queue.cc:260] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.711681 3440 raft_consensus.cc:695] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Becoming Leader. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Running, Role: LEADER
I20250109 20:55:23.712188 3440 consensus_queue.cc:237] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.715039 3431 ts_tablet_manager.cc:1428] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250109 20:55:23.717154 3461 raft_consensus.cc:513] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } }
I20250109 20:55:23.721647 3431 tablet_bootstrap.cc:492] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:23.724092 3431 tablet_bootstrap.cc:654] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.727072 3461 leader_election.cc:290] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Requested vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.727607 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd203495bb5452e9187ee9df32ca046" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653"
I20250109 20:55:23.727922 3181 raft_consensus.cc:3054] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.727942 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd203495bb5452e9187ee9df32ca046" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
I20250109 20:55:23.728168 3316 raft_consensus.cc:3054] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.731058 3316 raft_consensus.cc:2463] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:23.731626 2973 leader_election.cc:304] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters:
I20250109 20:55:23.722229 3460 raft_consensus.cc:2798] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:23.732499 3181 raft_consensus.cc:2463] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:23.732949 3465 raft_consensus.cc:2798] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:23.733124 3465 raft_consensus.cc:695] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Becoming Leader. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Running, Role: LEADER
I20250109 20:55:23.722492 3432 ts_tablet_manager.cc:1428] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.083s user 0.039s sys 0.033s
I20250109 20:55:23.734740 3432 tablet_bootstrap.cc:492] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.736148 3431 tablet_bootstrap.cc:492] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.736316 3431 ts_tablet_manager.cc:1397] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20250109 20:55:23.736649 3432 tablet_bootstrap.cc:654] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.733536 3465 consensus_queue.cc:237] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } }
I20250109 20:55:23.737114 3431 raft_consensus.cc:357] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.737318 3431 raft_consensus.cc:383] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.737397 3431 raft_consensus.cc:738] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.737668 3431 consensus_queue.cc:260] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.740094 3431 ts_tablet_manager.cc:1428] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250109 20:55:23.743845 2841 catalog_manager.cc:2149] created HMS catalog entry for table [id=fd4270c63e4c4ba1a145c0e996e16fb3]
I20250109 20:55:23.748131 3460 raft_consensus.cc:695] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Becoming Leader. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Running, Role: LEADER
I20250109 20:55:23.748512 3460 consensus_queue.cc:237] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.758616 3433 tablet_bootstrap.cc:492] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.758805 3433 ts_tablet_manager.cc:1397] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.066s user 0.025s sys 0.022s
I20250109 20:55:23.759613 3433 raft_consensus.cc:357] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.759840 3433 raft_consensus.cc:383] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.759922 3433 raft_consensus.cc:738] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.760186 3433 consensus_queue.cc:260] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.763197 3433 ts_tablet_manager.cc:1428] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250109 20:55:23.766230 3433 tablet_bootstrap.cc:492] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.767889 3433 tablet_bootstrap.cc:654] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.768831 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 (DEFAULT_TABLE table=default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.769390 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.774884 3019 tablet_service.cc:1467] Processing CreateTablet for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 (DEFAULT_TABLE table=default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.775204 3433 tablet_bootstrap.cc:492] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.775403 3433 ts_tablet_manager.cc:1397] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20250109 20:55:23.775555 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.776226 3433 raft_consensus.cc:357] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.776499 3433 raft_consensus.cc:383] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.776613 3433 raft_consensus.cc:738] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.776902 3433 consensus_queue.cc:260] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.777472 3433 ts_tablet_manager.cc:1428] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:55:23.777774 3433 tablet_bootstrap.cc:492] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.779485 3433 tablet_bootstrap.cc:654] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.780211 3161 tablet_service.cc:1467] Processing CreateTablet for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 (DEFAULT_TABLE table=default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.780710 3161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.793628 3433 tablet_bootstrap.cc:492] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.793813 3433 ts_tablet_manager.cc:1397] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20250109 20:55:23.794620 3433 raft_consensus.cc:357] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.794842 3433 raft_consensus.cc:383] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.794919 3433 raft_consensus.cc:738] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.795212 3433 consensus_queue.cc:260] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.798161 3433 ts_tablet_manager.cc:1428] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:55:23.803515 3431 tablet_bootstrap.cc:492] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:23.801064 2841 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56034:
table { table_name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2" } modify_external_catalogs: true
I20250109 20:55:23.804324 2841 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56034:
table { table_name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2" } modify_external_catalogs: true
I20250109 20:55:23.805306 3431 tablet_bootstrap.cc:654] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.805588 3433 tablet_bootstrap.cc:492] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.805526 3443 raft_consensus.cc:491] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:23.805816 3443 raft_consensus.cc:513] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.807898 3433 tablet_bootstrap.cc:654] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.808337 3443 leader_election.cc:290] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.816507 3432 tablet_bootstrap.cc:492] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.816723 3432 ts_tablet_manager.cc:1397] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.017s
I20250109 20:55:23.819545 3431 tablet_bootstrap.cc:492] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.819710 3431 ts_tablet_manager.cc:1397] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.013s
I20250109 20:55:23.820559 3431 raft_consensus.cc:357] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.820801 3431 raft_consensus.cc:383] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.820927 3431 raft_consensus.cc:738] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.821256 3431 consensus_queue.cc:260] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.820079 3432 raft_consensus.cc:357] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.822284 3432 raft_consensus.cc:383] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.822422 3432 raft_consensus.cc:738] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.822568 3433 tablet_bootstrap.cc:492] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:23.822762 3433 ts_tablet_manager.cc:1397] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20250109 20:55:23.823292 3432 consensus_queue.cc:260] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.823686 3433 raft_consensus.cc:357] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.823916 3433 raft_consensus.cc:383] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.824064 3433 raft_consensus.cc:738] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.824344 3433 consensus_queue.cc:260] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.825011 3437 raft_consensus.cc:491] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:23.825198 3437 raft_consensus.cc:513] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.825851 3433 ts_tablet_manager.cc:1428] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250109 20:55:23.826575 3465 raft_consensus.cc:491] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:23.826746 3465 raft_consensus.cc:513] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.827131 3437 leader_election.cc:290] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:23.827447 3465 leader_election.cc:290] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.828480 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af7427bf3674ed4b00d954366688451" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:23.828723 3181 raft_consensus.cc:2463] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:23.829046 3431 ts_tablet_manager.cc:1428] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20250109 20:55:23.829466 2971 leader_election.cc:304] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:23.831785 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af7427bf3674ed4b00d954366688451" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:23.832027 3316 raft_consensus.cc:2463] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:23.832719 3465 raft_consensus.cc:2798] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:23.832821 3465 raft_consensus.cc:491] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:23.832927 3465 raft_consensus.cc:3054] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.836112 3465 raft_consensus.cc:513] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.837206 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af7427bf3674ed4b00d954366688451" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653"
I20250109 20:55:23.837441 3181 raft_consensus.cc:3054] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.840430 3181 raft_consensus.cc:2463] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:23.841584 2971 leader_election.cc:304] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:23.841969 3467 raft_consensus.cc:2798] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:23.842139 3467 raft_consensus.cc:695] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Becoming Leader. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Running, Role: LEADER
I20250109 20:55:23.842505 3467 consensus_queue.cc:237] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.855543 3465 leader_election.cc:290] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Requested vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.856726 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d7677ecbb9460192ca4323d38b03ad" candidate_uuid: "cd7534821da847d0bf53f35562f4b653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:23.857159 3316 raft_consensus.cc:2388] T d8d7677ecbb9460192ca4323d38b03ad P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd7534821da847d0bf53f35562f4b653 in current term 1: Already voted for candidate 0d843776b4dc4cc194def9bd50945c29 in this term.
I20250109 20:55:23.863857 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af7427bf3674ed4b00d954366688451" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
I20250109 20:55:23.864135 3316 raft_consensus.cc:3054] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.867326 3316 raft_consensus.cc:2463] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:23.869098 3432 ts_tablet_manager.cc:1428] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.052s user 0.000s sys 0.003s
I20250109 20:55:23.869454 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae383e807bc4040b00d6c1afcdfdcfe" candidate_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:23.869750 3181 raft_consensus.cc:2463] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aab39ff46cf459e8ba528e5356ab3b6 in term 0.
I20250109 20:55:23.870424 3250 leader_election.cc:304] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:23.870993 3437 raft_consensus.cc:2798] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:23.871129 3437 raft_consensus.cc:491] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:23.871209 3437 raft_consensus.cc:3054] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.873013 3432 tablet_bootstrap.cc:492] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.875875 3437 raft_consensus.cc:513] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.876888 3432 tablet_bootstrap.cc:654] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.877136 3437 leader_election.cc:290] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 1 election: Requested vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:23.877822 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae383e807bc4040b00d6c1afcdfdcfe" candidate_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653"
I20250109 20:55:23.878108 3181 raft_consensus.cc:3054] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.880182 3040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d7677ecbb9460192ca4323d38b03ad" candidate_uuid: "cd7534821da847d0bf53f35562f4b653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d843776b4dc4cc194def9bd50945c29" is_pre_election: true
I20250109 20:55:23.880863 3039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae383e807bc4040b00d6c1afcdfdcfe" candidate_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d843776b4dc4cc194def9bd50945c29" is_pre_election: true
I20250109 20:55:23.881408 3039 raft_consensus.cc:2463] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aab39ff46cf459e8ba528e5356ab3b6 in term 0.
I20250109 20:55:23.882804 3116 leader_election.cc:304] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd7534821da847d0bf53f35562f4b653; no voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29
I20250109 20:55:23.883322 3443 raft_consensus.cc:3054] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.883533 3181 raft_consensus.cc:2463] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aab39ff46cf459e8ba528e5356ab3b6 in term 1.
I20250109 20:55:23.883977 3250 leader_election.cc:304] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:23.884303 3437 raft_consensus.cc:2798] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:23.886731 3443 raft_consensus.cc:2743] T d8d7677ecbb9460192ca4323d38b03ad P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:55:23.881974 3038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae383e807bc4040b00d6c1afcdfdcfe" candidate_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d843776b4dc4cc194def9bd50945c29"
I20250109 20:55:23.887391 3038 raft_consensus.cc:3054] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.891281 3432 tablet_bootstrap.cc:492] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.891396 3038 raft_consensus.cc:2463] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aab39ff46cf459e8ba528e5356ab3b6 in term 1.
I20250109 20:55:23.891593 3432 ts_tablet_manager.cc:1397] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20250109 20:55:23.892519 3432 raft_consensus.cc:357] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.892791 3432 raft_consensus.cc:383] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.892908 3432 raft_consensus.cc:738] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.893172 3432 consensus_queue.cc:260] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.893707 3432 ts_tablet_manager.cc:1428] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:55:23.894021 3432 tablet_bootstrap.cc:492] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.895027 3437 raft_consensus.cc:695] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Running, Role: LEADER
I20250109 20:55:23.895723 3432 tablet_bootstrap.cc:654] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.895550 3437 consensus_queue.cc:237] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.901064 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--webserver_interface=127.2.96.126
--webserver_port=37329
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:55:23.903031 3432 tablet_bootstrap.cc:492] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.906611 3432 ts_tablet_manager.cc:1397] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20250109 20:55:23.907471 3432 raft_consensus.cc:357] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.907678 3432 raft_consensus.cc:383] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.907778 3432 raft_consensus.cc:738] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.908048 3432 consensus_queue.cc:260] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.909760 3432 ts_tablet_manager.cc:1428] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250109 20:55:23.912307 2838 catalog_manager.cc:2149] created HMS catalog entry for table [id=c96a98e7e45749cb8ed5ffdfab329c31]
I20250109 20:55:23.932449 2838 catalog_manager.cc:5526] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65). New cstate: current_term: 1 leader_uuid: "0d843776b4dc4cc194def9bd50945c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } health_report { overall_health: HEALTHY } } }
I20250109 20:55:23.933429 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 0f95a0f62d3f4666adee4d1a51682f20 (DEFAULT_TABLE table=default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.933881 3296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f95a0f62d3f4666adee4d1a51682f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.938760 3019 tablet_service.cc:1467] Processing CreateTablet for tablet 0f95a0f62d3f4666adee4d1a51682f20 (DEFAULT_TABLE table=default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.939224 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f95a0f62d3f4666adee4d1a51682f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.946755 3161 tablet_service.cc:1467] Processing CreateTablet for tablet 0f95a0f62d3f4666adee4d1a51682f20 (DEFAULT_TABLE table=default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:23.947193 3161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f95a0f62d3f4666adee4d1a51682f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:23.954465 3432 tablet_bootstrap.cc:492] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:23.956137 3432 tablet_bootstrap.cc:654] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.960135 3443 raft_consensus.cc:491] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:23.960336 3443 raft_consensus.cc:513] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.961102 3443 leader_election.cc:290] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.961596 2838 catalog_manager.cc:5526] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65). New cstate: current_term: 1 leader_uuid: "0d843776b4dc4cc194def9bd50945c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } health_report { overall_health: UNKNOWN } } }
I20250109 20:55:23.962117 3038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a47d3e2c8ea4690bbdbe87936af9415" candidate_uuid: "cd7534821da847d0bf53f35562f4b653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d843776b4dc4cc194def9bd50945c29" is_pre_election: true
I20250109 20:55:23.962435 3038 raft_consensus.cc:2463] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7534821da847d0bf53f35562f4b653 in term 0.
I20250109 20:55:23.962530 3433 tablet_bootstrap.cc:492] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:23.962988 3116 leader_election.cc:304] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:23.963840 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a47d3e2c8ea4690bbdbe87936af9415" candidate_uuid: "cd7534821da847d0bf53f35562f4b653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:23.964085 3316 raft_consensus.cc:2463] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7534821da847d0bf53f35562f4b653 in term 0.
I20250109 20:55:23.964335 3433 tablet_bootstrap.cc:654] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.964850 3443 raft_consensus.cc:2798] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:23.964985 3443 raft_consensus.cc:491] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:23.965119 3443 raft_consensus.cc:3054] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.967259 3431 tablet_bootstrap.cc:492] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:23.967985 3443 raft_consensus.cc:513] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.969018 3431 tablet_bootstrap.cc:654] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:23.969414 3038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a47d3e2c8ea4690bbdbe87936af9415" candidate_uuid: "cd7534821da847d0bf53f35562f4b653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d843776b4dc4cc194def9bd50945c29"
I20250109 20:55:23.969652 3038 raft_consensus.cc:3054] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.972709 3038 raft_consensus.cc:2463] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7534821da847d0bf53f35562f4b653 in term 1.
I20250109 20:55:23.973304 3116 leader_election.cc:304] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:23.975263 2838 catalog_manager.cc:5526] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65). New cstate: current_term: 1 leader_uuid: "0d843776b4dc4cc194def9bd50945c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } health_report { overall_health: UNKNOWN } } }
I20250109 20:55:23.975730 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a47d3e2c8ea4690bbdbe87936af9415" candidate_uuid: "cd7534821da847d0bf53f35562f4b653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
I20250109 20:55:23.975848 3431 tablet_bootstrap.cc:492] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:23.975989 3316 raft_consensus.cc:3054] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:23.976038 3431 ts_tablet_manager.cc:1397] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20250109 20:55:23.976411 3432 tablet_bootstrap.cc:492] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:23.976616 3432 ts_tablet_manager.cc:1397] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250109 20:55:23.976902 3431 raft_consensus.cc:357] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.977152 3431 raft_consensus.cc:383] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.977253 3431 raft_consensus.cc:738] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.977515 3431 consensus_queue.cc:260] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.978847 3432 raft_consensus.cc:357] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.979120 3431 ts_tablet_manager.cc:1428] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:55:23.985371 3316 raft_consensus.cc:2463] T 8a47d3e2c8ea4690bbdbe87936af9415 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7534821da847d0bf53f35562f4b653 in term 1.
I20250109 20:55:23.986205 3443 leader_election.cc:290] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 1 election: Requested vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:23.986773 3443 raft_consensus.cc:2798] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:23.986976 3443 raft_consensus.cc:695] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [term 1 LEADER]: Becoming Leader. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Running, Role: LEADER
I20250109 20:55:23.986980 3432 raft_consensus.cc:383] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:23.987154 3432 raft_consensus.cc:738] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:23.987509 3432 consensus_queue.cc:260] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.987442 3443 consensus_queue.cc:237] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:23.992406 3432 ts_tablet_manager.cc:1428] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250109 20:55:23.999408 2838 catalog_manager.cc:5526] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67). New cstate: current_term: 1 leader_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } health_report { overall_health: HEALTHY } } }
I20250109 20:55:24.001829 3433 tablet_bootstrap.cc:492] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:24.002054 3433 ts_tablet_manager.cc:1397] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.006s
I20250109 20:55:24.002892 3433 raft_consensus.cc:357] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.003206 3433 raft_consensus.cc:383] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:24.003331 3433 raft_consensus.cc:738] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:24.003723 3433 consensus_queue.cc:260] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.008045 3460 raft_consensus.cc:491] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:24.008268 3460 raft_consensus.cc:513] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.009001 3460 leader_election.cc:290] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:24.010289 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f95a0f62d3f4666adee4d1a51682f20" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:24.010622 3316 raft_consensus.cc:2463] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:24.011091 2973 leader_election.cc:304] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters:
I20250109 20:55:24.012364 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f95a0f62d3f4666adee4d1a51682f20" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:24.012629 3181 raft_consensus.cc:2463] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:24.014158 3460 raft_consensus.cc:2798] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:24.014292 3460 raft_consensus.cc:491] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:24.014425 3460 raft_consensus.cc:3054] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.018127 3460 raft_consensus.cc:513] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.019815 3460 leader_election.cc:290] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Requested vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:24.020730 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f95a0f62d3f4666adee4d1a51682f20" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653"
I20250109 20:55:24.020967 3181 raft_consensus.cc:3054] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.023706 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f95a0f62d3f4666adee4d1a51682f20" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
I20250109 20:55:24.023983 3316 raft_consensus.cc:3054] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.024914 2838 catalog_manager.cc:5526] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65). New cstate: current_term: 1 leader_uuid: "0d843776b4dc4cc194def9bd50945c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } health_report { overall_health: UNKNOWN } } }
I20250109 20:55:24.028674 3181 raft_consensus.cc:2463] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:24.029616 3316 raft_consensus.cc:2463] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:24.030182 2973 leader_election.cc:304] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters:
I20250109 20:55:24.035673 3460 raft_consensus.cc:2798] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:24.037602 3460 raft_consensus.cc:695] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Becoming Leader. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Running, Role: LEADER
I20250109 20:55:24.037906 3460 consensus_queue.cc:237] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.039634 3433 ts_tablet_manager.cc:1428] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.037s user 0.004s sys 0.000s
I20250109 20:55:24.048702 2838 catalog_manager.cc:5526] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd7534821da847d0bf53f35562f4b653 (127.2.96.66). New cstate: current_term: 1 leader_uuid: "cd7534821da847d0bf53f35562f4b653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } health_report { overall_health: UNKNOWN } } }
I20250109 20:55:24.067149 2838 catalog_manager.cc:5526] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65). New cstate: current_term: 1 leader_uuid: "0d843776b4dc4cc194def9bd50945c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } health_report { overall_health: UNKNOWN } } }
I20250109 20:55:24.078892 2838 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_48885eaf759344b8a2b07e6b94fa13ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:24.093484 3460 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:24.093710 3460 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.094619 3460 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:24.095894 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eb8b3ef2b0414da0e9de1ba8c4a129" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:24.096176 3181 raft_consensus.cc:2463] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:24.096581 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eb8b3ef2b0414da0e9de1ba8c4a129" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:24.096822 3316 raft_consensus.cc:2463] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:24.097311 2973 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters:
I20250109 20:55:24.098033 3460 raft_consensus.cc:2798] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:24.098155 3460 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:24.098260 3460 raft_consensus.cc:3054] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.100997 3460 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.102185 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eb8b3ef2b0414da0e9de1ba8c4a129" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653"
I20250109 20:55:24.102429 3181 raft_consensus.cc:3054] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.105188 3181 raft_consensus.cc:2463] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:24.105685 2971 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:24.106001 3465 raft_consensus.cc:2798] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:24.106166 3465 raft_consensus.cc:695] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Becoming Leader. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Running, Role: LEADER
I20250109 20:55:24.106504 3465 consensus_queue.cc:237] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.109943 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eb8b3ef2b0414da0e9de1ba8c4a129" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
I20250109 20:55:24.110170 3316 raft_consensus.cc:3054] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.111601 3460 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Requested vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:24.112818 3316 raft_consensus.cc:2463] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
20:55:24.118 [HMS - ERROR - pool-6-thread-1] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_f7ca428ec2984ae4992e760dd6c2f8eb already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20250109 20:55:24.120230 2840 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=a39194c4a5034eaa80cdf0220b88d974]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_f7ca428ec2984ae4992e760dd6c2f8eb already exists)
I20250109 20:55:24.120645 2840 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56034 (request call id 19) took 1062 ms. Trace:
I20250109 20:55:24.120746 2840 rpcz_store.cc:276] 0109 20:55:23.057873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:55:23.058675 (+ 802us) service_pool.cc:225] Handling call
0109 20:55:23.230262 (+171587us) catalog_manager.cc:2076] Acquired catalog manager lock
0109 20:55:23.234522 (+ 4260us) catalog_manager.cc:2122] Created new table and tablet info
0109 20:55:24.120614 (+886092us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":677563,"Hive Metastore.run_cpu_time_us":919,"Hive Metastore.run_wall_time_us":207783}
I20250109 20:55:24.122418 2840 catalog_manager.cc:5526] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65). New cstate: current_term: 1 leader_uuid: "0d843776b4dc4cc194def9bd50945c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } health_report { overall_health: UNKNOWN } } }
I20250109 20:55:24.129645 2840 master_service.cc:946] ReplaceTablet: received request to replace tablet 7ae383e807bc4040b00d6c1afcdfdcfe from {username='slave'} at 127.0.0.1:56034
I20250109 20:55:24.136096 2840 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Sending DeleteTablet for 3 replicas of tablet 7ae383e807bc4040b00d6c1afcdfdcfe
I20250109 20:55:24.136561 2840 catalog_manager.cc:6596] ReplaceTablet: tablet 7ae383e807bc4040b00d6c1afcdfdcfe deleted and replaced by tablet dc461d653b944163bcf564e04efb74e2
I20250109 20:55:24.136893 2840 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56034:
name: "default.Table_20e2171c365c4c3e8aafa501e51dae5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:24.140117 3019 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae383e807bc4040b00d6c1afcdfdcfe with delete_type TABLET_DATA_DELETED (replaced by tablet dc461d653b944163bcf564e04efb74e2) from {username='slave'} at 127.0.0.1:45928
I20250109 20:55:24.140723 3161 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae383e807bc4040b00d6c1afcdfdcfe with delete_type TABLET_DATA_DELETED (replaced by tablet dc461d653b944163bcf564e04efb74e2) from {username='slave'} at 127.0.0.1:40368
I20250109 20:55:24.142066 3296 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae383e807bc4040b00d6c1afcdfdcfe with delete_type TABLET_DATA_DELETED (replaced by tablet dc461d653b944163bcf564e04efb74e2) from {username='slave'} at 127.0.0.1:49628
I20250109 20:55:24.148021 3020 tablet_service.cc:1467] Processing CreateTablet for tablet dc461d653b944163bcf564e04efb74e2 (DEFAULT_TABLE table=default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:24.148535 3020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc461d653b944163bcf564e04efb74e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:24.148631 3295 tablet_service.cc:1467] Processing CreateTablet for tablet dc461d653b944163bcf564e04efb74e2 (DEFAULT_TABLE table=default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:24.149051 3295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc461d653b944163bcf564e04efb74e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:24.152472 3160 tablet_service.cc:1467] Processing CreateTablet for tablet dc461d653b944163bcf564e04efb74e2 (DEFAULT_TABLE table=default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:55:24.152842 3160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc461d653b944163bcf564e04efb74e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:55:24.152937 3498 tablet_replica.cc:331] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: stopping tablet replica
I20250109 20:55:24.153252 3498 raft_consensus.cc:2238] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:55:24.153810 3498 pending_rounds.cc:62] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: Trying to abort 1 pending ops.
I20250109 20:55:24.154036 3498 pending_rounds.cc:69] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112524283481681920 op_type: NO_OP noop_request { }
I20250109 20:55:24.154253 3498 raft_consensus.cc:2883] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:55:24.154407 3498 raft_consensus.cc:2267] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.159263 3498 ts_tablet_manager.cc:1905] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.159412 3431 tablet_bootstrap.cc:492] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6: Bootstrap starting.
I20250109 20:55:24.161216 3431 tablet_bootstrap.cc:654] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:24.162248 3432 tablet_bootstrap.cc:492] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29: Bootstrap starting.
I20250109 20:55:24.164532 3432 tablet_bootstrap.cc:654] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:24.165620 3496 tablet_replica.cc:331] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: stopping tablet replica
I20250109 20:55:24.165725 3433 tablet_bootstrap.cc:492] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653: Bootstrap starting.
I20250109 20:55:24.165835 3496 raft_consensus.cc:2238] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:24.166039 3496 raft_consensus.cc:2267] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.167769 3433 tablet_bootstrap.cc:654] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653: Neither blocks nor log segments found. Creating new log.
I20250109 20:55:24.168556 3496 ts_tablet_manager.cc:1905] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.171716 3497 tablet_replica.cc:331] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: stopping tablet replica
I20250109 20:55:24.172071 3497 raft_consensus.cc:2238] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:24.172537 3497 raft_consensus.cc:2267] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.174000 3497 ts_tablet_manager.cc:1905] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.175269 3432 tablet_bootstrap.cc:492] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29: No bootstrap required, opened a new log
I20250109 20:55:24.175248 3498 ts_tablet_manager.cc:1918] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:55:24.175411 3498 log.cc:1198] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal/wals/7ae383e807bc4040b00d6c1afcdfdcfe
I20250109 20:55:24.175565 3432 ts_tablet_manager.cc:1397] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20250109 20:55:24.175882 3498 ts_tablet_manager.cc:1939] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting consensus metadata
I20250109 20:55:24.176584 3432 raft_consensus.cc:357] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.176862 3432 raft_consensus.cc:383] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:24.177002 3432 raft_consensus.cc:738] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Initialized, Role: FOLLOWER
I20250109 20:55:24.177063 2832 catalog_manager.cc:4872] TS 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259): tablet 7ae383e807bc4040b00d6c1afcdfdcfe (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]) successfully deleted
I20250109 20:55:24.177335 3432 consensus_queue.cc:260] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.177980 3432 ts_tablet_manager.cc:1428] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:55:24.186230 3431 tablet_bootstrap.cc:492] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6: No bootstrap required, opened a new log
I20250109 20:55:24.186547 3431 ts_tablet_manager.cc:1397] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.009s
I20250109 20:55:24.187364 3431 raft_consensus.cc:357] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.187786 3431 raft_consensus.cc:383] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:24.188110 3431 raft_consensus.cc:738] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Initialized, Role: FOLLOWER
I20250109 20:55:24.188895 3431 consensus_queue.cc:260] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
F20250109 20:55:24.189162 2902 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cbd0b67985f24940af88114907c635eb
*** Check failure stack trace: ***
I20250109 20:55:24.191057 3431 ts_tablet_manager.cc:1428] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
@ 0x7fafba42bdcd google::LogMessage::Fail() at ??:0
I20250109 20:55:24.192593 3433 tablet_bootstrap.cc:492] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653: No bootstrap required, opened a new log
I20250109 20:55:24.192751 3433 ts_tablet_manager.cc:1397] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.007s
@ 0x7fafba42fb93 google::LogMessage::SendToLog() at ??:0
I20250109 20:55:24.193588 3433 raft_consensus.cc:357] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.193846 3433 raft_consensus.cc:383] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:55:24.193974 3433 raft_consensus.cc:738] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Initialized, Role: FOLLOWER
I20250109 20:55:24.194211 3433 consensus_queue.cc:260] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
@ 0x7fafba42b7cc google::LogMessage::Flush() at ??:0
I20250109 20:55:24.196820 3433 ts_tablet_manager.cc:1428] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
@ 0x7fafba42ba11 google::LogMessage::~LogMessage() at ??:0
I20250109 20:55:24.199343 3496 ts_tablet_manager.cc:1918] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:24.199458 3496 log.cc:1198] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/wals/7ae383e807bc4040b00d6c1afcdfdcfe
I20250109 20:55:24.199766 3496 ts_tablet_manager.cc:1939] T 7ae383e807bc4040b00d6c1afcdfdcfe P 0d843776b4dc4cc194def9bd50945c29: Deleting consensus metadata
I20250109 20:55:24.200879 2831 catalog_manager.cc:4872] TS 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): tablet 7ae383e807bc4040b00d6c1afcdfdcfe (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]) successfully deleted
I20250109 20:55:24.203365 3497 ts_tablet_manager.cc:1918] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:24.203473 3497 log.cc:1198] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal/wals/7ae383e807bc4040b00d6c1afcdfdcfe
I20250109 20:55:24.203931 3497 ts_tablet_manager.cc:1939] T 7ae383e807bc4040b00d6c1afcdfdcfe P cd7534821da847d0bf53f35562f4b653: Deleting consensus metadata
I20250109 20:55:24.204869 2830 catalog_manager.cc:4872] TS cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): tablet 7ae383e807bc4040b00d6c1afcdfdcfe (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]) successfully deleted
@ 0x7fafd05ce6de kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fafd05cd3c1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fafd05d44dc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fafd05d4359 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
@ 0x7fafbd1890be kudu::Thread::SuperviseThread() at ??:0
@ 0x7fafc056c6db start_thread at ??:0
@ 0x7fafb92dc71f clone at ??:0
I20250109 20:55:24.279642 3460 raft_consensus.cc:491] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:24.279848 3460 raft_consensus.cc:513] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.280701 3460 leader_election.cc:290] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:24.281364 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc461d653b944163bcf564e04efb74e2" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:24.281596 3181 raft_consensus.cc:2463] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:24.282115 2971 leader_election.cc:304] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:24.282109 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc461d653b944163bcf564e04efb74e2" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:24.282341 3316 raft_consensus.cc:2463] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 0.
I20250109 20:55:24.282428 3460 raft_consensus.cc:2798] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:24.282527 3460 raft_consensus.cc:491] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:24.282600 3460 raft_consensus.cc:3054] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.285691 3460 raft_consensus.cc:513] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.286811 3437 raft_consensus.cc:491] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:24.287072 3437 raft_consensus.cc:513] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.287184 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc461d653b944163bcf564e04efb74e2" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
I20250109 20:55:24.287439 3316 raft_consensus.cc:3054] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.287806 3437 leader_election.cc:290] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:24.288877 3038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e6865a502d40bb986e4df59db8425c" candidate_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d843776b4dc4cc194def9bd50945c29" is_pre_election: true
I20250109 20:55:24.289076 3038 raft_consensus.cc:2463] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aab39ff46cf459e8ba528e5356ab3b6 in term 0.
I20250109 20:55:24.289538 3251 leader_election.cc:304] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters:
I20250109 20:55:24.289848 3437 raft_consensus.cc:2798] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:55:24.289937 3437 raft_consensus.cc:491] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:24.290041 3437 raft_consensus.cc:3054] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.290288 3316 raft_consensus.cc:2463] T dc461d653b944163bcf564e04efb74e2 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:24.290787 3180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e6865a502d40bb986e4df59db8425c" candidate_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:24.291065 3180 raft_consensus.cc:2463] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aab39ff46cf459e8ba528e5356ab3b6 in term 0.
I20250109 20:55:24.291771 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc461d653b944163bcf564e04efb74e2" candidate_uuid: "0d843776b4dc4cc194def9bd50945c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653"
I20250109 20:55:24.291952 3181 raft_consensus.cc:3054] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.292706 3437 raft_consensus.cc:513] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.293830 3180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e6865a502d40bb986e4df59db8425c" candidate_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653"
I20250109 20:55:24.294013 3180 raft_consensus.cc:3054] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.294790 3181 raft_consensus.cc:2463] T dc461d653b944163bcf564e04efb74e2 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d843776b4dc4cc194def9bd50945c29 in term 1.
I20250109 20:55:24.295696 2973 leader_election.cc:304] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, 0d843776b4dc4cc194def9bd50945c29; no voters:
I20250109 20:55:24.296017 3465 raft_consensus.cc:2798] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:24.296152 3465 raft_consensus.cc:695] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Becoming Leader. State: Replica: 0d843776b4dc4cc194def9bd50945c29, State: Running, Role: LEADER
I20250109 20:55:24.296459 3465 consensus_queue.cc:237] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:24.297251 3180 raft_consensus.cc:2463] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aab39ff46cf459e8ba528e5356ab3b6 in term 1.
I20250109 20:55:24.297777 3038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e6865a502d40bb986e4df59db8425c" candidate_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d843776b4dc4cc194def9bd50945c29"
I20250109 20:55:24.297945 3038 raft_consensus.cc:3054] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:55:24.297921 3250 leader_election.cc:304] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, cd7534821da847d0bf53f35562f4b653; no voters:
I20250109 20:55:24.298254 3484 raft_consensus.cc:2798] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:55:24.300048 3437 leader_election.cc:290] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 1 election: Requested vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:24.300688 3038 raft_consensus.cc:2463] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aab39ff46cf459e8ba528e5356ab3b6 in term 1.
I20250109 20:55:24.302472 3460 leader_election.cc:290] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [CANDIDATE]: Term 1 election: Requested vote from peers cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:24.310768 3484 raft_consensus.cc:695] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0aab39ff46cf459e8ba528e5356ab3b6, State: Running, Role: LEADER
I20250109 20:55:24.311172 3484 consensus_queue.cc:237] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
W20250109 20:55:24.418936 3487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:24.419389 3487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:24.419569 3487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:24.435793 3487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:24.435940 3487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:24.436029 3487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:24.436197 3487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250109 20:55:24.438138 3371 connection.cc:537] client connection to 127.2.96.124:46741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:24.439067 3371 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.96.124:46741)
W20250109 20:55:24.439334 3371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.124:46741)
W20250109 20:55:24.439884 3371 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.96.124:46741)
W20250109 20:55:24.440945 3371 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.96.124:46741)
W20250109 20:55:24.443454 3478 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:55:24.438892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.96.124:46741 (local address 127.2.96.67:46057)
0109 20:55:24.439411 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0109 20:55:24.439431 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0109 20:55:24.439550 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 20:55:24.442576 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":377}
I20250109 20:55:24.449935 3487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.96.126
--webserver_port=37329
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:24.451931 3487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:24.453840 3487 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:24.465090 3523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:24.465598 3524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:24.466710 3526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:24.467154 3487 server_base.cc:1034] running on GCE node
I20250109 20:55:24.468183 3487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:24.469671 3487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:24.470882 3487 hybrid_clock.cc:648] HybridClock initialized: now 1736456124470821 us; error 38 us; skew 500 ppm
I20250109 20:55:24.471210 3487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:24.473448 3487 webserver.cc:458] Webserver started at http://127.2.96.126:37329/ using document root <none> and password file <none>
I20250109 20:55:24.474170 3487 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:24.474330 3487 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:24.480042 3487 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20250109 20:55:24.482995 3532 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:24.484390 3487 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250109 20:55:24.484573 3487 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
uuid: "b9908c8a5d72498890b9058df83a1b5e"
format_stamp: "Formatted at 2025-01-09 20:55:19 on dist-test-slave-thwl"
I20250109 20:55:24.485687 3487 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:24.506006 3487 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:24.506878 3487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:24.507122 3487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:24.525300 3487 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.126:37541
I20250109 20:55:24.525311 3583 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.126:37541 every 8 connection(s)
I20250109 20:55:24.526805 3487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
I20250109 20:55:24.532778 3584 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:55:24.534117 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 3487
I20250109 20:55:24.536851 3584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap starting.
I20250109 20:55:24.548102 3584 log.cc:826] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:24.557040 3584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:55:24.557560 3584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap complete.
I20250109 20:55:24.563791 3584 raft_consensus.cc:357] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:24.564538 3584 raft_consensus.cc:738] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Initialized, Role: FOLLOWER
I20250109 20:55:24.565115 3584 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:24.565929 3589 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:24.566267 3589 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:24.567114 3584 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:24.577803 3603 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:24.577916 3603 catalog_manager.cc:1553] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: loading cluster ID for follower catalog manager: success
I20250109 20:55:24.581508 3603 catalog_manager.cc:1575] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: acquiring CA information for follower catalog manager: success
I20250109 20:55:24.588549 3603 catalog_manager.cc:1603] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:24.607522 3487 master_runner.cc:186] Error getting master registration for 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111),
I20250109 20:55:24.612673 3487 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:24.656108 3608 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95304c9e18024004b2277a53318ad77e)
I20250109 20:55:24.656311 3608 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:24.657146 3608 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:24.659963 2762 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:24.664873 3559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "b9908c8a5d72498890b9058df83a1b5e" is_pre_election: true
I20250109 20:55:24.665633 3559 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d84981b978e3495fb36dc33f09cc825a in term 2.
I20250109 20:55:24.666178 2762 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e
I20250109 20:55:24.666587 3608 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250109 20:55:24.666692 3608 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 95304c9e18024004b2277a53318ad77e)
I20250109 20:55:24.666790 3608 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 3 FOLLOWER]: Advancing to term 4
I20250109 20:55:24.668882 3608 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:24.669441 3608 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 4 election: Requested vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:24.669924 3559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "b9908c8a5d72498890b9058df83a1b5e"
I20250109 20:55:24.670157 3559 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 2 FOLLOWER]: Advancing to term 4
W20250109 20:55:24.670954 2762 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:24.674355 3559 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d84981b978e3495fb36dc33f09cc825a in term 4.
I20250109 20:55:24.674782 2762 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e
I20250109 20:55:24.675120 3608 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 4 FOLLOWER]: Leader election won for term 4
I20250109 20:55:24.675562 3608 raft_consensus.cc:695] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 4 LEADER]: Becoming Leader. State: Replica: d84981b978e3495fb36dc33f09cc825a, State: Running, Role: LEADER
I20250109 20:55:24.675969 3608 consensus_queue.cc:237] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 3.36, Last appended by leader: 36, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:24.677273 3612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84981b978e3495fb36dc33f09cc825a. Latest consensus state: current_term: 4 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:24.677599 3612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:24.678299 3613 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:24.681548 3613 catalog_manager.cc:670] Loaded metadata for table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]
I20250109 20:55:24.682094 3613 catalog_manager.cc:670] Loaded metadata for table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]
I20250109 20:55:24.682586 3613 catalog_manager.cc:670] Loaded metadata for table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]
I20250109 20:55:24.683056 3613 catalog_manager.cc:670] Loaded metadata for table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]
I20250109 20:55:24.683478 3613 catalog_manager.cc:670] Loaded metadata for table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]
I20250109 20:55:24.683893 3613 catalog_manager.cc:670] Loaded metadata for table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]
I20250109 20:55:24.684317 3613 catalog_manager.cc:670] Loaded metadata for table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]
I20250109 20:55:24.684759 3613 catalog_manager.cc:670] Loaded metadata for table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]
I20250109 20:55:24.685204 3613 catalog_manager.cc:670] Loaded metadata for table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]
I20250109 20:55:24.687636 3613 tablet_loader.cc:96] loaded metadata for tablet 0af7427bf3674ed4b00d954366688451 (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06])
I20250109 20:55:24.688155 3613 tablet_loader.cc:96] loaded metadata for tablet 0f95a0f62d3f4666adee4d1a51682f20 (table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31])
I20250109 20:55:24.688663 3613 tablet_loader.cc:96] loaded metadata for tablet 0fd203495bb5452e9187ee9df32ca046 (table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc])
I20250109 20:55:24.689615 3613 tablet_loader.cc:96] loaded metadata for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 (table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3])
I20250109 20:55:24.690045 3613 tablet_loader.cc:96] loaded metadata for tablet 8a47d3e2c8ea4690bbdbe87936af9415 (table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc])
I20250109 20:55:24.690438 3613 tablet_loader.cc:96] loaded metadata for tablet 93e6865a502d40bb986e4df59db8425c (table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702])
I20250109 20:55:24.690927 3613 tablet_loader.cc:96] loaded metadata for tablet b685d1fbcd744586b8e0a3024e27a4ff (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c])
I20250109 20:55:24.691345 3613 tablet_loader.cc:96] loaded metadata for tablet d8d7677ecbb9460192ca4323d38b03ad (table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869])
I20250109 20:55:24.691735 3613 tablet_loader.cc:96] loaded metadata for tablet dc461d653b944163bcf564e04efb74e2 (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec])
I20250109 20:55:24.692344 3613 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:24.693775 3613 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:24.693885 3613 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:24.695622 3613 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:24.696877 3613 catalog_manager.cc:5910] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Loaded TSK: 0
I20250109 20:55:24.698050 3613 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:24.699319 3613 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:24.700546 3613 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:24.703864 3227 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:24.707660 3548 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cd7534821da847d0bf53f35562f4b653" instance_seqno: 1736456122119526) as {username='slave'} at 127.2.96.66:48395; Asking this server to re-register.
I20250109 20:55:24.708384 3227 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:24.709992 3548 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:24.720039 2775 master_service.cc:946] ReplaceTablet: received request to replace tablet 0fd203495bb5452e9187ee9df32ca046 from {username='slave'} at 127.0.0.1:56454
I20250109 20:55:24.722565 2766 master_service.cc:946] ReplaceTablet: received request to replace tablet 0fd203495bb5452e9187ee9df32ca046 from {username='slave'} at 127.0.0.1:56454
I20250109 20:55:24.721442 2774 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56454:
name: "default.Table_aabd822000834e5b9e47ffbb942b54ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:24.722137 2772 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56454:
name: "default.Table_0d2dc89f52bc4afc8d3de54485a3de62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:24.722536 2769 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56454:
table { table_name: "default.Table_d2f3663d9ed949609d925f9bb37f7633" } modify_external_catalogs: true
I20250109 20:55:24.722716 2767 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56454:
name: "default.Table_e672267cd18e4969b28037e40d7bc5cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:24.727975 2769 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56454:
table { table_name: "default.Table_d2f3663d9ed949609d925f9bb37f7633" } modify_external_catalogs: true
I20250109 20:55:24.722267 2771 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56454:
table { table_name: "default.Table_f5f640c97c72444396c24fbb34b9ec18" } modify_external_catalogs: true
I20250109 20:55:24.720662 2773 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56454:
name: "default.Table_22c1f44dd42c4195a778637ee4b876ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:24.728376 2771 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56454:
table { table_name: "default.Table_f5f640c97c72444396c24fbb34b9ec18" } modify_external_catalogs: true
I20250109 20:55:24.723568 2768 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56454:
name: "default.Table_66e7788f7db54a2ea368c19b84195df5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:24.723387 2770 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56454:
name: "default.Table_2132f850190647e6af0ace1b48daa3ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250109 20:55:24.735150 2948 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cbd0b67985f24940af88114907c635eb
*** Check failure stack trace: ***
I20250109 20:55:24.737726 3559 raft_consensus.cc:1270] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Refusing update from remote peer d84981b978e3495fb36dc33f09cc825a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 38. (index mismatch)
W20250109 20:55:24.738202 2762 consensus_peers.cc:487] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a -> Peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Couldn't send request to peer 95304c9e18024004b2277a53318ad77e. Status: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 20:55:24.738747 3612 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f67e1d1bdcd google::LogMessage::Fail() at ??:0
@ 0x7f67e1d1fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f67e1d1b7cc google::LogMessage::Flush() at ??:0
@ 0x7f67e1d1ba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f67f7ebe6de kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f67f7ebd3c1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f67f7ec44dc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f67f7ec4359 std::_Function_handler<>::_M_invoke() at ??:0
I20250109 20:55:24.770833 3465 consensus_queue.cc:1035] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:24.772099 3589 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84981b978e3495fb36dc33f09cc825a. Latest consensus state: current_term: 4 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:24.772382 3589 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:24.773388 3621 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:24.773677 3621 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:24.777897 3621 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:24.778129 3621 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:24.780225 3465 consensus_queue.cc:1035] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0xa71ecd std::function<>::operator()() at ??:?
I20250109 20:55:24.789655 3362 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:24.789925 3362 heartbeater.cc:502] Master 127.2.96.126:37541 was elected leader, sending a full tablet report...
I20250109 20:55:24.795270 3548 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" instance_seqno: 1736456122600579) as {username='slave'} at 127.2.96.67:41713; Asking this server to re-register.
I20250109 20:55:24.795403 2775 catalog_manager.cc:5813] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Sending DeleteTablet for 3 replicas of tablet 0fd203495bb5452e9187ee9df32ca046
I20250109 20:55:24.796377 3362 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:24.797379 2775 catalog_manager.cc:6596] ReplaceTablet: tablet 0fd203495bb5452e9187ee9df32ca046 deleted and replaced by tablet 1010e1aeb5d140bf9f29d386bc195ed7
I20250109 20:55:24.798494 2766 master_service.cc:946] ReplaceTablet: received request to replace tablet 0af7427bf3674ed4b00d954366688451 from {username='slave'} at 127.0.0.1:56454
I20250109 20:55:24.799222 3548 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:24.808604 3019 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd203495bb5452e9187ee9df32ca046 with delete_type TABLET_DATA_DELETED (replaced by tablet 1010e1aeb5d140bf9f29d386bc195ed7) from {username='slave'} at 127.0.0.1:42346
I20250109 20:55:24.809099 3612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:24.809439 3612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:24.812309 2775 master_service.cc:946] ReplaceTablet: received request to replace tablet 93e6865a502d40bb986e4df59db8425c from {username='slave'} at 127.0.0.1:56454
@ 0x7f67e4a790be kudu::Thread::SuperviseThread() at ??:0
I20250109 20:55:24.823221 3635 tablet_replica.cc:331] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: stopping tablet replica
I20250109 20:55:24.827705 2766 catalog_manager.cc:5813] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Sending DeleteTablet for 3 replicas of tablet 0af7427bf3674ed4b00d954366688451
I20250109 20:55:24.828253 2766 catalog_manager.cc:6596] ReplaceTablet: tablet 0af7427bf3674ed4b00d954366688451 deleted and replaced by tablet c5339f2d6a1547c3aad25d50455eeab2
I20250109 20:55:24.828994 3019 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af7427bf3674ed4b00d954366688451 with delete_type TABLET_DATA_DELETED (replaced by tablet c5339f2d6a1547c3aad25d50455eeab2) from {username='slave'} at 127.0.0.1:42346
I20250109 20:55:24.829190 3635 raft_consensus.cc:2238] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:55:24.829784 3467 consensus_queue.cc:1035] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250109 20:55:24.831507 3635 raft_consensus.cc:2267] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.832499 3465 consensus_queue.cc:1035] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:24.836396 3635 ts_tablet_manager.cc:1905] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.854197 3635 ts_tablet_manager.cc:1918] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:55:24.854373 3635 log.cc:1198] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/wals/0fd203495bb5452e9187ee9df32ca046
I20250109 20:55:24.859267 3635 ts_tablet_manager.cc:1939] T 0fd203495bb5452e9187ee9df32ca046 P 0d843776b4dc4cc194def9bd50945c29: Deleting consensus metadata
I20250109 20:55:24.860606 2761 catalog_manager.cc:4872] TS 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): tablet 0fd203495bb5452e9187ee9df32ca046 (table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]) successfully deleted
@ 0x7f67e7e5c6db start_thread at ??:0
I20250109 20:55:24.864548 3228 heartbeater.cc:502] Master 127.2.96.125:37271 was elected leader, sending a full tablet report...
I20250109 20:55:24.866891 3635 tablet_replica.cc:331] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: stopping tablet replica
I20250109 20:55:24.867129 3635 raft_consensus.cc:2238] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:55:24.867513 3635 pending_rounds.cc:62] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Trying to abort 1 pending ops.
I20250109 20:55:24.867724 3635 pending_rounds.cc:69] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112524283261853696 op_type: NO_OP noop_request { }
I20250109 20:55:24.867898 3635 raft_consensus.cc:2883] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:55:24.868152 3086 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
W20250109 20:55:24.868088 2762 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.96.126:37541, user_credentials={real_user=slave}} blocked reactor thread for 50229.7us
I20250109 20:55:24.870465 3363 heartbeater.cc:502] Master 127.2.96.125:37271 was elected leader, sending a full tablet report...
I20250109 20:55:24.876133 3087 heartbeater.cc:502] Master 127.2.96.125:37271 was elected leader, sending a full tablet report...
I20250109 20:55:24.877178 3467 consensus_queue.cc:1035] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:24.877987 3161 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd203495bb5452e9187ee9df32ca046 with delete_type TABLET_DATA_DELETED (replaced by tablet 1010e1aeb5d140bf9f29d386bc195ed7) from {username='slave'} at 127.0.0.1:52754
I20250109 20:55:24.878659 2775 catalog_manager.cc:5813] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Sending DeleteTablet for 3 replicas of tablet 93e6865a502d40bb986e4df59db8425c
I20250109 20:55:24.878846 3160 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af7427bf3674ed4b00d954366688451 with delete_type TABLET_DATA_DELETED (replaced by tablet c5339f2d6a1547c3aad25d50455eeab2) from {username='slave'} at 127.0.0.1:52754
I20250109 20:55:24.879240 2775 catalog_manager.cc:6596] ReplaceTablet: tablet 93e6865a502d40bb986e4df59db8425c deleted and replaced by tablet 07334099f3184a179000213a1839dddf
I20250109 20:55:24.879566 2775 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56454:
table { table_name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2" } modify_external_catalogs: true
I20250109 20:55:24.879791 2775 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56454:
table { table_name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2" } modify_external_catalogs: true
I20250109 20:55:24.880965 3160 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e6865a502d40bb986e4df59db8425c with delete_type TABLET_DATA_DELETED (replaced by tablet 07334099f3184a179000213a1839dddf) from {username='slave'} at 127.0.0.1:52754
I20250109 20:55:24.881989 3019 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e6865a502d40bb986e4df59db8425c with delete_type TABLET_DATA_DELETED (replaced by tablet 07334099f3184a179000213a1839dddf) from {username='slave'} at 127.0.0.1:42346
I20250109 20:55:24.887144 3296 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af7427bf3674ed4b00d954366688451 with delete_type TABLET_DATA_DELETED (replaced by tablet c5339f2d6a1547c3aad25d50455eeab2) from {username='slave'} at 127.0.0.1:51518
I20250109 20:55:24.887463 3295 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd203495bb5452e9187ee9df32ca046 with delete_type TABLET_DATA_DELETED (replaced by tablet 1010e1aeb5d140bf9f29d386bc195ed7) from {username='slave'} at 127.0.0.1:51518
I20250109 20:55:24.892630 3295 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e6865a502d40bb986e4df59db8425c with delete_type TABLET_DATA_DELETED (replaced by tablet 07334099f3184a179000213a1839dddf) from {username='slave'} at 127.0.0.1:51518
I20250109 20:55:24.898568 3648 tablet_replica.cc:331] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: stopping tablet replica
I20250109 20:55:24.899094 3648 raft_consensus.cc:2238] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:24.899331 3648 raft_consensus.cc:2267] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.902172 3460 consensus_queue.cc:1035] T d8d7677ecbb9460192ca4323d38b03ad P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:24.903064 3648 ts_tablet_manager.cc:1905] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.909431 3644 tablet_replica.cc:331] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: stopping tablet replica
@ 0x7f67e0bcc71f clone at ??:0
I20250109 20:55:24.925206 3465 consensus_queue.cc:1035] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:24.925563 3655 consensus_queue.cc:1035] T b685d1fbcd744586b8e0a3024e27a4ff P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:24.931901 3616 mvcc.cc:204] Tried to move back new op lower bound from 7112524286892953600 to 7112524279949639680. Current Snapshot: MvccSnapshot[applied={T|T < 7112524286892953600 or (T in {7112524286892953600})}]
I20250109 20:55:24.937222 3635 raft_consensus.cc:2267] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.937920 3635 ts_tablet_manager.cc:1905] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.945043 3635 ts_tablet_manager.cc:1918] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:55:24.945187 3635 log.cc:1198] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/wals/0af7427bf3674ed4b00d954366688451
I20250109 20:55:24.945638 3635 ts_tablet_manager.cc:1939] T 0af7427bf3674ed4b00d954366688451 P 0d843776b4dc4cc194def9bd50945c29: Deleting consensus metadata
I20250109 20:55:24.946159 3635 tablet_replica.cc:331] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: stopping tablet replica
I20250109 20:55:24.946379 3635 raft_consensus.cc:2238] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:24.946580 3635 raft_consensus.cc:2267] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.947113 3635 ts_tablet_manager.cc:1905] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.951467 3627 consensus_queue.cc:1035] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:24.952950 3635 ts_tablet_manager.cc:1918] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:24.953086 3635 log.cc:1198] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/wals/93e6865a502d40bb986e4df59db8425c
I20250109 20:55:24.953569 3635 ts_tablet_manager.cc:1939] T 93e6865a502d40bb986e4df59db8425c P 0d843776b4dc4cc194def9bd50945c29: Deleting consensus metadata
I20250109 20:55:24.959566 3640 consensus_queue.cc:1035] T 8a47d3e2c8ea4690bbdbe87936af9415 P cd7534821da847d0bf53f35562f4b653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:24.961769 3644 raft_consensus.cc:2238] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:24.962551 3644 raft_consensus.cc:2267] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.963513 3644 ts_tablet_manager.cc:1905] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.971407 3644 ts_tablet_manager.cc:1918] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:55:24.971529 3644 log.cc:1198] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal/wals/0fd203495bb5452e9187ee9df32ca046
I20250109 20:55:24.971944 3644 ts_tablet_manager.cc:1939] T 0fd203495bb5452e9187ee9df32ca046 P cd7534821da847d0bf53f35562f4b653: Deleting consensus metadata
I20250109 20:55:24.972427 3644 tablet_replica.cc:331] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: stopping tablet replica
I20250109 20:55:24.972625 3644 raft_consensus.cc:2238] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:24.972774 3644 raft_consensus.cc:2267] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.973282 3644 ts_tablet_manager.cc:1905] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.979813 3549 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" instance_seqno: 1736456121613668) as {username='slave'} at 127.2.96.65:36179; Asking this server to re-register.
I20250109 20:55:24.981463 3086 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:24.982491 3644 ts_tablet_manager.cc:1918] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:24.982894 3648 ts_tablet_manager.cc:1918] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:24.983032 3648 log.cc:1198] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal/wals/0af7427bf3674ed4b00d954366688451
I20250109 20:55:24.983450 3648 ts_tablet_manager.cc:1939] T 0af7427bf3674ed4b00d954366688451 P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting consensus metadata
I20250109 20:55:24.983716 3549 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:24.984840 3648 tablet_replica.cc:331] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: stopping tablet replica
I20250109 20:55:24.985044 3648 raft_consensus.cc:2238] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:24.985224 3648 raft_consensus.cc:2267] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.985951 3648 ts_tablet_manager.cc:1905] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.988551 3644 log.cc:1198] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal/wals/0af7427bf3674ed4b00d954366688451
I20250109 20:55:24.988946 3644 ts_tablet_manager.cc:1939] T 0af7427bf3674ed4b00d954366688451 P cd7534821da847d0bf53f35562f4b653: Deleting consensus metadata
I20250109 20:55:24.989917 3644 tablet_replica.cc:331] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: stopping tablet replica
I20250109 20:55:24.990096 3644 raft_consensus.cc:2238] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:24.990260 3644 raft_consensus.cc:2267] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.990793 3644 ts_tablet_manager.cc:1905] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.992544 3648 ts_tablet_manager.cc:1918] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:55:24.992671 3648 log.cc:1198] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal/wals/0fd203495bb5452e9187ee9df32ca046
I20250109 20:55:24.993059 3648 ts_tablet_manager.cc:1939] T 0fd203495bb5452e9187ee9df32ca046 P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting consensus metadata
I20250109 20:55:24.994081 3648 tablet_replica.cc:331] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: stopping tablet replica
I20250109 20:55:24.994283 3648 raft_consensus.cc:2238] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:55:24.994916 3648 pending_rounds.cc:62] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: Trying to abort 1 pending ops.
I20250109 20:55:24.994980 3648 pending_rounds.cc:69] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112524285182980096 op_type: NO_OP noop_request { }
I20250109 20:55:24.995141 3648 raft_consensus.cc:2883] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:55:24.995232 3648 raft_consensus.cc:2267] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:24.996001 3648 ts_tablet_manager.cc:1905] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:24.996920 3644 ts_tablet_manager.cc:1918] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:55:24.997025 3644 log.cc:1198] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal/wals/93e6865a502d40bb986e4df59db8425c
I20250109 20:55:24.997416 3644 ts_tablet_manager.cc:1939] T 93e6865a502d40bb986e4df59db8425c P cd7534821da847d0bf53f35562f4b653: Deleting consensus metadata
I20250109 20:55:25.005370 3648 ts_tablet_manager.cc:1918] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:55:25.005528 3648 log.cc:1198] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-2/wal/wals/93e6865a502d40bb986e4df59db8425c
I20250109 20:55:25.005944 3648 ts_tablet_manager.cc:1939] T 93e6865a502d40bb986e4df59db8425c P 0aab39ff46cf459e8ba528e5356ab3b6: Deleting consensus metadata
I20250109 20:55:25.061978 3460 consensus_queue.cc:1035] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:25.067972 3654 consensus_queue.cc:1035] T dc461d653b944163bcf564e04efb74e2 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:25.069087 3654 consensus_queue.cc:1035] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:55:25.089150 3460 consensus_queue.cc:1035] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:55:25.153952 3115 connection.cc:537] server connection from 127.0.0.1:52754 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:25.154625 3368 connection.cc:537] client connection to 127.2.96.125:37271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:25.156131 3252 connection.cc:537] server connection from 127.0.0.1:51518 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:25.156504 2971 connection.cc:537] server connection from 127.0.0.1:42346 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:25.157820 2970 connection.cc:537] client connection to 127.2.96.125:37271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:25.154326 3114 connection.cc:537] client connection to 127.2.96.125:37271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250109 20:55:25.373289 3669 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:25.373529 3669 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:25.375417 3669 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:25.376505 3536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
W20250109 20:55:25.377784 3533 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:25.378257 3536 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:25.378438 3536 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e; no voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:25.378835 3669 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250109 20:55:26.036067 3681 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:26.036257 3681 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:26.036912 3681 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:26.038949 3536 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
W20250109 20:55:26.039086 3533 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:26.039247 3533 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e; no voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:26.039521 3681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250109 20:55:26.593876 3687 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:26.594067 3687 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:26.594764 3687 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:26.596596 3533 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:26.596751 3536 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:26.596854 3536 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e; no voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:26.597116 3687 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250109 20:55:27.360529 3701 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:27.360816 3701 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:27.361474 3701 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:27.363309 3536 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
W20250109 20:55:27.363528 3533 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:27.363641 3533 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e; no voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:27.363879 3701 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250109 20:55:27.404980 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.125:37271
--webserver_interface=127.2.96.125
--webserver_port=39189
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250109 20:55:27.733997 3705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:27.734325 3705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:27.734477 3705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:27.742586 3705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:27.742712 3705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:27.742786 3705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:27.742861 3705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:27.753393 3705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.125:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.96.125
--webserver_port=39189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:27.755204 3705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:27.756899 3705 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:27.767565 3715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:27.768620 3717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:27.768887 3714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:27.770027 3705 server_base.cc:1034] running on GCE node
I20250109 20:55:27.770874 3705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:27.772336 3705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:27.773573 3705 hybrid_clock.cc:648] HybridClock initialized: now 1736456127773507 us; error 41 us; skew 500 ppm
I20250109 20:55:27.773949 3705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:27.776605 3705 webserver.cc:458] Webserver started at http://127.2.96.125:39189/ using document root <none> and password file <none>
I20250109 20:55:27.777338 3705 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:27.777485 3705 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:27.783167 3705 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250109 20:55:27.785944 3723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:27.787240 3705 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250109 20:55:27.787416 3705 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
uuid: "d84981b978e3495fb36dc33f09cc825a"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:27.788257 3705 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:27.818770 3705 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:27.819680 3705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:27.819962 3705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:27.837846 3705 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.125:37271
I20250109 20:55:27.837870 3774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.125:37271 every 8 connection(s)
I20250109 20:55:27.839416 3705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
I20250109 20:55:27.843199 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 3705
I20250109 20:55:27.846781 3775 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:55:27.851441 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap starting.
I20250109 20:55:27.865908 3775 log.cc:826] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:27.896677 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=27 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:55:27.897187 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap complete.
I20250109 20:55:27.903841 3775 raft_consensus.cc:357] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:27.904568 3775 raft_consensus.cc:738] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84981b978e3495fb36dc33f09cc825a, State: Initialized, Role: FOLLOWER
I20250109 20:55:27.905144 3775 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 4.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:27.905879 3780 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:27.906141 3780 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:27.906966 3775 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:27.922142 3794 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:27.922255 3794 catalog_manager.cc:1553] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: loading cluster ID for follower catalog manager: success
I20250109 20:55:27.929056 3794 catalog_manager.cc:1575] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: acquiring CA information for follower catalog manager: success
I20250109 20:55:27.932477 3794 catalog_manager.cc:1603] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:27.949445 3705 master_runner.cc:186] Error getting master registration for 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111),
I20250109 20:55:27.954056 3705 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:28.062952 3799 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:28.063160 3799 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:28.063864 3799 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:28.066708 3536 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:28.071780 3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 5 candidate_status { last_received { term: 4 index: 40 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
I20250109 20:55:28.072500 3750 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 4.
I20250109 20:55:28.073072 3533 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e
I20250109 20:55:28.073374 3799 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250109 20:55:28.073496 3799 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:28.073593 3799 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 4 FOLLOWER]: Advancing to term 5
I20250109 20:55:28.075652 3799 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:28.076326 3799 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 election: Requested vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:28.076814 3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 5 candidate_status { last_received { term: 4 index: 40 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a"
I20250109 20:55:28.077047 3750 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 4 FOLLOWER]: Advancing to term 5
W20250109 20:55:28.077800 3536 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:28.081413 3750 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 5.
I20250109 20:55:28.081882 3533 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e
I20250109 20:55:28.082218 3799 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 FOLLOWER]: Leader election won for term 5
I20250109 20:55:28.082720 3799 raft_consensus.cc:695] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 LEADER]: Becoming Leader. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Running, Role: LEADER
I20250109 20:55:28.083269 3799 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 4.40, Last appended by leader: 40, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:28.085330 3803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 5 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:28.085667 3803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:28.086354 3805 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:28.089395 3805 catalog_manager.cc:670] Loaded metadata for table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]
I20250109 20:55:28.090029 3805 catalog_manager.cc:670] Loaded metadata for table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]
I20250109 20:55:28.090548 3805 catalog_manager.cc:670] Loaded metadata for table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]
I20250109 20:55:28.091008 3805 catalog_manager.cc:670] Loaded metadata for table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]
I20250109 20:55:28.091487 3805 catalog_manager.cc:670] Loaded metadata for table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]
I20250109 20:55:28.091918 3805 catalog_manager.cc:670] Loaded metadata for table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]
I20250109 20:55:28.092336 3805 catalog_manager.cc:670] Loaded metadata for table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]
I20250109 20:55:28.092711 3805 catalog_manager.cc:670] Loaded metadata for table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]
I20250109 20:55:28.093032 3805 catalog_manager.cc:670] Loaded metadata for table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]
I20250109 20:55:28.094813 3805 tablet_loader.cc:96] loaded metadata for tablet 07334099f3184a179000213a1839dddf (table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702])
I20250109 20:55:28.095580 3805 tablet_loader.cc:96] loaded metadata for tablet 0f95a0f62d3f4666adee4d1a51682f20 (table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31])
I20250109 20:55:28.096074 3805 tablet_loader.cc:96] loaded metadata for tablet 1010e1aeb5d140bf9f29d386bc195ed7 (table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc])
I20250109 20:55:28.096688 3805 tablet_loader.cc:96] loaded metadata for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 (table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3])
I20250109 20:55:28.097059 3805 tablet_loader.cc:96] loaded metadata for tablet 8a47d3e2c8ea4690bbdbe87936af9415 (table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc])
I20250109 20:55:28.097662 3805 tablet_loader.cc:96] loaded metadata for tablet b685d1fbcd744586b8e0a3024e27a4ff (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c])
I20250109 20:55:28.097997 3805 tablet_loader.cc:96] loaded metadata for tablet c5339f2d6a1547c3aad25d50455eeab2 (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06])
I20250109 20:55:28.098346 3805 tablet_loader.cc:96] loaded metadata for tablet d8d7677ecbb9460192ca4323d38b03ad (table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869])
I20250109 20:55:28.098686 3805 tablet_loader.cc:96] loaded metadata for tablet dc461d653b944163bcf564e04efb74e2 (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec])
I20250109 20:55:28.099267 3805 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:28.100399 3805 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:28.100476 3805 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:28.101743 3805 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:28.102922 3805 catalog_manager.cc:5910] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Loaded TSK: 0
I20250109 20:55:28.103849 3805 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:28.104805 3805 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:28.105634 3805 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:28.175606 3363 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:28.175866 3363 heartbeater.cc:502] Master 127.2.96.125:37271 was elected leader, sending a full tablet report...
I20250109 20:55:28.179167 3740 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" instance_seqno: 1736456122600579) as {username='slave'} at 127.2.96.67:56413; Asking this server to re-register.
I20250109 20:55:28.180065 3363 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:28.181200 3542 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56888:
table { table_name: "default.Table_edccb187ef7149b1a1d158edc0924690" } new_table_name: "default.Table_1b53d8836f984435902fbc1c6a9073b1" modify_external_catalogs: true
I20250109 20:55:28.182458 3739 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:28.183017 3549 master_service.cc:946] ReplaceTablet: received request to replace tablet d8d7677ecbb9460192ca4323d38b03ad from {username='slave'} at 127.0.0.1:56888
I20250109 20:55:28.185756 3547 master_service.cc:946] ReplaceTablet: received request to replace tablet dc461d653b944163bcf564e04efb74e2 from {username='slave'} at 127.0.0.1:56888
I20250109 20:55:28.187191 3546 master_service.cc:946] ReplaceTablet: received request to replace tablet dc461d653b944163bcf564e04efb74e2 from {username='slave'} at 127.0.0.1:56888
I20250109 20:55:28.187973 3545 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56888:
name: "default.Table_20e2171c365c4c3e8aafa501e51dae5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:28.188768 3544 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56888:
name: "default.Table_48885eaf759344b8a2b07e6b94fa13ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250109 20:55:28.189654 3602 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cbd0b67985f24940af88114907c635eb
*** Check failure stack trace: ***
I20250109 20:55:28.190011 3541 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56888:
table { table_name: "default.Table_320b149b7d80477bb465bb9b3351d0be" } new_table_name: "default.Table_9c2e38f90afc4b9dbf223d20ffa931fb" modify_external_catalogs: true
I20250109 20:55:28.190582 3540 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56888:
table { table_name: "default.Table_e27d0bbbd6a445f099a3be8e238bf9c8" } new_table_name: "default.Table_c66ec467f4264b9bbd248d748ba125c0" modify_external_catalogs: true
I20250109 20:55:28.189498 3543 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56888:
name: "default.Table_e992250ce97b4de3ac9cf07ff61775a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:28.191725 3548 master_service.cc:946] ReplaceTablet: received request to replace tablet dc461d653b944163bcf564e04efb74e2 from {username='slave'} at 127.0.0.1:56888
@ 0x7fb0399a3dcd google::LogMessage::Fail() at ??:0
@ 0x7fb0399a7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb0399a37cc google::LogMessage::Flush() at ??:0
@ 0x7fb0399a3a11 google::LogMessage::~LogMessage() at ??:0
I20250109 20:55:28.207660 3750 raft_consensus.cc:1270] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Refusing update from remote peer b9908c8a5d72498890b9058df83a1b5e: Log matching property violated. Preceding OpId in replica: term: 4 index: 40. Preceding OpId from leader: term: 5 index: 42. (index mismatch)
I20250109 20:55:28.208647 3799 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20250109 20:55:28.209426 3536 consensus_peers.cc:487] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e -> Peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Couldn't send request to peer 95304c9e18024004b2277a53318ad77e. Status: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fb04fb466de kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fb04fb453c1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fb04fb4c4dc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250109 20:55:28.235074 3780 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 5 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:28.235436 3780 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fb04fb4c359 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
@ 0x7fb03c7010be kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb03fae46db start_thread at ??:0
I20250109 20:55:28.250811 3087 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:28.251044 3087 heartbeater.cc:502] Master 127.2.96.125:37271 was elected leader, sending a full tablet report...
@ 0x7fb03885471f clone at ??:0
I20250109 20:55:28.258074 3740 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" instance_seqno: 1736456121613668) as {username='slave'} at 127.2.96.65:49567; Asking this server to re-register.
I20250109 20:55:28.258563 3780 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:28.258800 3780 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:28.260933 3087 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:28.262856 3740 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:28.266851 3228 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:28.267038 3228 heartbeater.cc:502] Master 127.2.96.125:37271 was elected leader, sending a full tablet report...
I20250109 20:55:28.269582 3740 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cd7534821da847d0bf53f35562f4b653" instance_seqno: 1736456122119526) as {username='slave'} at 127.2.96.66:54653; Asking this server to re-register.
I20250109 20:55:28.270337 3228 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:28.272257 3740 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:55:28.465301 3727 connection.cc:537] server connection from 127.0.0.1:37318 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:28.465955 3117 connection.cc:537] client connection to 127.2.96.126:37541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:28.466181 3371 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.96.126:37541)
W20250109 20:55:28.466466 3371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:28.466691 3371 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:28.466867 3371 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.96.126:37541)
I20250109 20:55:28.612234 3362 heartbeater.cc:502] Master 127.2.96.126:37541 was elected leader, sending a full tablet report...
I20250109 20:55:28.745883 3780 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:28.746116 3780 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:28.748188 3780 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:28.748886 3727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:28.750182 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:28.750353 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:28.750478 3727 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:28.750784 3780 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250109 20:55:29.496101 3833 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:29.496315 3833 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:29.497128 3833 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:29.499493 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:29.499719 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:29.499823 3727 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:29.500080 3833 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250109 20:55:30.283550 3838 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:30.283731 3838 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:30.284343 3838 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:30.286635 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:30.286813 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:30.286927 3727 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:30.287179 3838 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250109 20:55:30.938683 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--webserver_interface=127.2.96.126
--webserver_port=37329
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:55:31.062847 3843 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:31.063036 3843 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:31.063782 3843 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:31.067234 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:31.067461 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:31.067545 3727 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:31.067804 3843 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250109 20:55:31.288481 3842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:31.288862 3842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:31.289006 3842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:31.297370 3842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:31.297518 3842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:31.297605 3842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:31.297701 3842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:31.307711 3842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.96.126
--webserver_port=37329
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:31.309485 3842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:31.311394 3842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:31.322222 3852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:31.323222 3854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:31.323799 3851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:31.324827 3842 server_base.cc:1034] running on GCE node
I20250109 20:55:31.325625 3842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:31.327122 3842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:31.328341 3842 hybrid_clock.cc:648] HybridClock initialized: now 1736456131328226 us; error 96 us; skew 500 ppm
I20250109 20:55:31.328665 3842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:31.337889 3842 webserver.cc:458] Webserver started at http://127.2.96.126:37329/ using document root <none> and password file <none>
I20250109 20:55:31.338734 3842 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:31.338852 3842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:31.344166 3842 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 20:55:31.346982 3860 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:31.348390 3842 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250109 20:55:31.348577 3842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
uuid: "b9908c8a5d72498890b9058df83a1b5e"
format_stamp: "Formatted at 2025-01-09 20:55:19 on dist-test-slave-thwl"
I20250109 20:55:31.349439 3842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:31.371361 3842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:31.372246 3842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:31.372515 3842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:31.391203 3842 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.126:37541
I20250109 20:55:31.391222 3912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.126:37541 every 8 connection(s)
I20250109 20:55:31.392859 3842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
I20250109 20:55:31.398603 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 3842
I20250109 20:55:31.399369 3913 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:55:31.403750 3913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap starting.
I20250109 20:55:31.415848 3913 log.cc:826] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:31.446484 3913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=41 ignored=0} inserts{seen=27 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250109 20:55:31.447047 3913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap complete.
I20250109 20:55:31.453366 3913 raft_consensus.cc:357] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:31.456434 3913 raft_consensus.cc:738] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Initialized, Role: FOLLOWER
I20250109 20:55:31.457218 3913 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 5.43, Last appended by leader: 43, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:31.458029 3918 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:31.458267 3918 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:31.459012 3913 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:31.470351 3932 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:31.470467 3932 catalog_manager.cc:1553] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: loading cluster ID for follower catalog manager: success
I20250109 20:55:31.474494 3932 catalog_manager.cc:1575] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: acquiring CA information for follower catalog manager: success
I20250109 20:55:31.482481 3932 catalog_manager.cc:1603] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:31.494413 3086 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:31.495577 3227 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:31.499320 3878 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cd7534821da847d0bf53f35562f4b653" instance_seqno: 1736456122119526) as {username='slave'} at 127.2.96.66:44807; Asking this server to re-register.
I20250109 20:55:31.499445 3875 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" instance_seqno: 1736456121613668) as {username='slave'} at 127.2.96.65:33339; Asking this server to re-register.
I20250109 20:55:31.500201 3227 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:31.501132 3086 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:31.502251 3875 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:31.502338 3878 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:31.518327 3842 master_runner.cc:186] Error getting master registration for 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111),
I20250109 20:55:31.523137 3842 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:31.627692 3362 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:31.627944 3362 heartbeater.cc:502] Master 127.2.96.126:37541 was elected leader, sending a full tablet report...
I20250109 20:55:31.630044 3875 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" instance_seqno: 1736456122600579) as {username='slave'} at 127.2.96.67:43505; Asking this server to re-register.
I20250109 20:55:31.630735 3362 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:31.632203 3875 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:32.014097 3938 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:32.014370 3938 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:32.016170 3938 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:32.016785 3749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
W20250109 20:55:32.016853 3864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:32.017027 3749 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 5.
I20250109 20:55:32.017572 3861 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters:
I20250109 20:55:32.017923 3938 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250109 20:55:32.018059 3938 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:32.018193 3938 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 5 FOLLOWER]: Advancing to term 6
W20250109 20:55:32.018303 3864 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:32.021514 3938 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:32.022094 3938 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 6 election: Requested vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:32.022526 3749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a"
I20250109 20:55:32.022742 3749 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 5 FOLLOWER]: Advancing to term 6
W20250109 20:55:32.023347 3864 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:32.024737 3749 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 6.
I20250109 20:55:32.025120 3861 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e
I20250109 20:55:32.025408 3938 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 6 FOLLOWER]: Leader election won for term 6
I20250109 20:55:32.025758 3938 raft_consensus.cc:695] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 6 LEADER]: Becoming Leader. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Running, Role: LEADER
I20250109 20:55:32.026168 3938 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 5.43, Last appended by leader: 43, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:32.028820 3940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 6 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:32.029107 3940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:32.548094 3749 raft_consensus.cc:1270] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Refusing update from remote peer b9908c8a5d72498890b9058df83a1b5e: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 6 index: 44. (index mismatch)
I20250109 20:55:32.548892 3946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250109 20:55:32.550904 3947 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 6 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:32.551162 3947 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:32.553593 3947 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:32.553812 3947 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:32.555414 3946 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:32.555641 3946 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: LEADER
W20250109 20:55:32.557200 3864 consensus_peers.cc:487] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e -> Peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Couldn't send request to peer 95304c9e18024004b2277a53318ad77e. Status: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 20:55:32.675364 3942 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:32.677924 3942 catalog_manager.cc:670] Loaded metadata for table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]
I20250109 20:55:32.678369 3942 catalog_manager.cc:670] Loaded metadata for table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]
I20250109 20:55:32.678691 3942 catalog_manager.cc:670] Loaded metadata for table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]
I20250109 20:55:32.678982 3942 catalog_manager.cc:670] Loaded metadata for table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]
I20250109 20:55:32.679252 3942 catalog_manager.cc:670] Loaded metadata for table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]
I20250109 20:55:32.679533 3942 catalog_manager.cc:670] Loaded metadata for table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]
I20250109 20:55:32.679819 3942 catalog_manager.cc:670] Loaded metadata for table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]
I20250109 20:55:32.680114 3942 catalog_manager.cc:670] Loaded metadata for table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]
I20250109 20:55:32.680395 3942 catalog_manager.cc:670] Loaded metadata for table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]
I20250109 20:55:32.682011 3942 tablet_loader.cc:96] loaded metadata for tablet 07334099f3184a179000213a1839dddf (table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702])
I20250109 20:55:32.682585 3942 tablet_loader.cc:96] loaded metadata for tablet 0f95a0f62d3f4666adee4d1a51682f20 (table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31])
I20250109 20:55:32.682968 3942 tablet_loader.cc:96] loaded metadata for tablet 1010e1aeb5d140bf9f29d386bc195ed7 (table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc])
I20250109 20:55:32.683542 3942 tablet_loader.cc:96] loaded metadata for tablet 7df6e6361c074dc3984dbe5d92b1205d (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec])
I20250109 20:55:32.683774 3942 tablet_loader.cc:96] loaded metadata for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 (table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3])
I20250109 20:55:32.684022 3942 tablet_loader.cc:96] loaded metadata for tablet 8a47d3e2c8ea4690bbdbe87936af9415 (table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc])
I20250109 20:55:32.684407 3942 tablet_loader.cc:96] loaded metadata for tablet b685d1fbcd744586b8e0a3024e27a4ff (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c])
I20250109 20:55:32.684620 3942 tablet_loader.cc:96] loaded metadata for tablet c17492c45ef04ace967a0cd0df3a8eaf (table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869])
I20250109 20:55:32.684811 3942 tablet_loader.cc:96] loaded metadata for tablet c5339f2d6a1547c3aad25d50455eeab2 (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06])
I20250109 20:55:32.685479 3942 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:32.686489 3942 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:32.686558 3942 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:32.687736 3942 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:32.688608 3942 catalog_manager.cc:5910] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Loaded TSK: 0
I20250109 20:55:32.689445 3942 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:32.690357 3942 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:32.691285 3942 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:32.715523 3876 master_service.cc:946] ReplaceTablet: received request to replace tablet 7ae383e807bc4040b00d6c1afcdfdcfe from {username='slave'} at 127.0.0.1:57658
I20250109 20:55:32.716210 3875 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57658:
name: "default.Table_5dcc9f9689084615b4c47a34fdd1a329"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:32.716504 3872 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57658:
name: "default.Table_66e7788f7db54a2ea368c19b84195df5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:32.716939 3869 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57658:
name: "default.Table_48885eaf759344b8a2b07e6b94fa13ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:32.717386 3874 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57658:
table { table_name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2" } modify_external_catalogs: true
I20250109 20:55:32.717656 3874 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57658:
table { table_name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2" } modify_external_catalogs: true
I20250109 20:55:32.717717 3871 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57658:
table { table_name: "default.Table_d2f3663d9ed949609d925f9bb37f7633" } modify_external_catalogs: true
I20250109 20:55:32.718600 3878 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57658:
table { table_name: "default.Table_84db4e769e42410f985d1d5f5cbe527a" } new_table_name: "default.Table_a2481e08ab1c4f2193462e658e9dbecc" modify_external_catalogs: true
I20250109 20:55:32.718601 3871 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57658:
table { table_name: "default.Table_d2f3663d9ed949609d925f9bb37f7633" } modify_external_catalogs: true
I20250109 20:55:32.719048 3868 master_service.cc:946] ReplaceTablet: received request to replace tablet dc461d653b944163bcf564e04efb74e2 from {username='slave'} at 127.0.0.1:57658
I20250109 20:55:32.718909 3870 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57658:
name: "default.Table_aabd822000834e5b9e47ffbb942b54ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:32.719473 3873 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57658:
name: "default.Table_e672267cd18e4969b28037e40d7bc5cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:32.719874 3868 master_service.cc:946] ReplaceTablet: received request to replace tablet dc461d653b944163bcf564e04efb74e2 from {username='slave'} at 127.0.0.1:57658
I20250109 20:55:32.718910 3876 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57658:
name: "default.Table_20e2171c365c4c3e8aafa501e51dae5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:32.720358 3868 master_service.cc:946] ReplaceTablet: received request to replace tablet dc461d653b944163bcf564e04efb74e2 from {username='slave'} at 127.0.0.1:57658
I20250109 20:55:32.720793 3868 master_service.cc:946] ReplaceTablet: received request to replace tablet d8d7677ecbb9460192ca4323d38b03ad from {username='slave'} at 127.0.0.1:57658
I20250109 20:55:32.721370 3868 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57658:
name: "default.Table_e992250ce97b4de3ac9cf07ff61775a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250109 20:55:32.723682 3931 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cbd0b67985f24940af88114907c635eb
*** Check failure stack trace: ***
@ 0x7f9f98d29dcd google::LogMessage::Fail() at ??:0
@ 0x7f9f98d2db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9f98d297cc google::LogMessage::Flush() at ??:0
@ 0x7f9f98d29a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9faeecc6de kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f9faeecb3c1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f9faeed24dc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f9faeed2359 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
@ 0x7f9f9ba870be kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9f9ee6a6db start_thread at ??:0
@ 0x7f9f97bda71f clone at ??:0
W20250109 20:55:32.925021 3371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:32.925429 3371 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:32.925773 3371 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:32.928272 3371 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.96.126:37541)
I20250109 20:55:32.955461 3705 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:55:32.955618 3705 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:33.054960 3973 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:33.055253 3973 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:33.055905 3973 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:33.058022 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:33.058198 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:33.058303 3727 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:33.058634 3973 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250109 20:55:33.762856 3978 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:33.763051 3978 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:33.763782 3978 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:33.764871 3727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250109 20:55:33.766264 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:33.766486 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:33.766597 3727 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:33.766827 3978 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250109 20:55:34.360307 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.124:46741
--webserver_interface=127.2.96.124
--webserver_port=36029
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:55:34.483966 3984 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:34.484227 3984 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:34.484917 3984 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:34.488301 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:34.488474 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:34.488562 3727 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:34.488834 3984 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250109 20:55:34.703318 3982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:34.703716 3982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:34.703845 3982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:34.711762 3982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:34.711889 3982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:34.711966 3982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:34.712041 3982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:34.721714 3982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.124:46741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.96.124
--webserver_port=36029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:34.723397 3982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:34.725085 3982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:34.735282 3992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:34.735787 3995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:34.737012 3982 server_base.cc:1034] running on GCE node
W20250109 20:55:34.737844 3993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:34.738778 3982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:34.740221 3982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:34.741470 3982 hybrid_clock.cc:648] HybridClock initialized: now 1736456134741401 us; error 42 us; skew 500 ppm
I20250109 20:55:34.741784 3982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:34.744117 3982 webserver.cc:458] Webserver started at http://127.2.96.124:36029/ using document root <none> and password file <none>
I20250109 20:55:34.744814 3982 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:34.744930 3982 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:34.749985 3982 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20250109 20:55:34.752889 4001 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:34.754179 3982 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250109 20:55:34.754357 3982 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
uuid: "95304c9e18024004b2277a53318ad77e"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:34.755266 3982 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:34.781565 3982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:34.782471 3982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:34.782754 3982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:34.801365 3982 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.124:46741
I20250109 20:55:34.801389 4052 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.124:46741 every 8 connection(s)
I20250109 20:55:34.802910 3982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/info.pb
I20250109 20:55:34.809934 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 3982
I20250109 20:55:34.810128 4053 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:55:34.814229 4053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Bootstrap starting.
I20250109 20:55:34.826882 4053 log.cc:826] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:34.855324 4053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=24 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:55:34.855885 4053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Bootstrap complete.
I20250109 20:55:34.862340 4053 raft_consensus.cc:357] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:34.863193 4053 raft_consensus.cc:738] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95304c9e18024004b2277a53318ad77e, State: Initialized, Role: FOLLOWER
I20250109 20:55:34.863845 4053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 3.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:34.864646 4057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:34.864956 4057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:34.865696 4053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:34.877539 4071 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:34.877650 4071 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: loading cluster ID for follower catalog manager: success
I20250109 20:55:34.881255 4071 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: acquiring CA information for follower catalog manager: success
I20250109 20:55:34.883999 4071 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:34.887660 3982 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:55:34.896759 3982 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:35.126955 3361 heartbeater.cc:346] Connected to a master server at 127.2.96.124:46741
I20250109 20:55:35.130674 4018 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" instance_seqno: 1736456122600579) as {username='slave'} at 127.2.96.67:50005; Asking this server to re-register.
I20250109 20:55:35.131583 3361 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:35.133725 4018 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:35.262028 3085 heartbeater.cc:346] Connected to a master server at 127.2.96.124:46741
I20250109 20:55:35.263898 4018 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" instance_seqno: 1736456121613668) as {username='slave'} at 127.2.96.65:36079; Asking this server to re-register.
I20250109 20:55:35.264467 3085 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:35.265684 4018 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:35.298578 4057 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:35.298882 4057 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:35.301098 4057 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:35.302117 4005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:35.308341 4005 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:35.309162 3226 heartbeater.cc:346] Connected to a master server at 127.2.96.124:46741
I20250109 20:55:35.311110 4018 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cd7534821da847d0bf53f35562f4b653" instance_seqno: 1736456122119526) as {username='slave'} at 127.2.96.66:41015; Asking this server to re-register.
I20250109 20:55:35.311720 3226 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:35.312481 3749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
I20250109 20:55:35.312788 3749 raft_consensus.cc:2363] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95304c9e18024004b2277a53318ad77e for earlier term 4. Current term is 6.
I20250109 20:55:35.313010 4018 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:35.313683 4002 leader_election.cc:400] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 4 pre-election: Vote denied by peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95304c9e18024004b2277a53318ad77e for earlier term 4. Current term is 6.
I20250109 20:55:35.313761 4002 leader_election.cc:403] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250109 20:55:35.314065 4057 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 3 FOLLOWER]: Advancing to term 6
I20250109 20:55:35.317937 4057 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 6 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95304c9e18024004b2277a53318ad77e for earlier term 4. Current term is 6.
I20250109 20:55:35.531054 4079 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:35.531268 4079 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:35.532191 4079 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:35.535089 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:35.539034 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 7 candidate_status { last_received { term: 6 index: 44 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e" is_pre_election: true
I20250109 20:55:35.539510 4028 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d84981b978e3495fb36dc33f09cc825a in term 6.
I20250109 20:55:35.540099 3727 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a; no voters: b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:35.540501 4079 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250109 20:55:35.540661 4079 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:35.540764 4079 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 6 FOLLOWER]: Advancing to term 7
I20250109 20:55:35.542984 4079 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:35.543679 4079 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 election: Requested vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:35.544739 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 7 candidate_status { last_received { term: 6 index: 44 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e"
I20250109 20:55:35.545010 4028 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 6 FOLLOWER]: Advancing to term 7
W20250109 20:55:35.545464 3727 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:35.547482 4028 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d84981b978e3495fb36dc33f09cc825a in term 7.
I20250109 20:55:35.548234 3727 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a; no voters: b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:35.548722 4079 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 7 FOLLOWER]: Leader election won for term 7
I20250109 20:55:35.549196 4079 raft_consensus.cc:695] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 7 LEADER]: Becoming Leader. State: Replica: d84981b978e3495fb36dc33f09cc825a, State: Running, Role: LEADER
I20250109 20:55:35.549661 4079 consensus_queue.cc:237] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 6.44, Last appended by leader: 44, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:35.551535 4083 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84981b978e3495fb36dc33f09cc825a. Latest consensus state: current_term: 7 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:35.551923 4083 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:35.552615 4085 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:35.555366 4085 catalog_manager.cc:670] Loaded metadata for table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]
I20250109 20:55:35.555902 4085 catalog_manager.cc:670] Loaded metadata for table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]
I20250109 20:55:35.556291 4085 catalog_manager.cc:670] Loaded metadata for table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]
I20250109 20:55:35.556658 4085 catalog_manager.cc:670] Loaded metadata for table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]
I20250109 20:55:35.557026 4085 catalog_manager.cc:670] Loaded metadata for table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]
I20250109 20:55:35.557395 4085 catalog_manager.cc:670] Loaded metadata for table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]
I20250109 20:55:35.557788 4085 catalog_manager.cc:670] Loaded metadata for table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]
I20250109 20:55:35.558147 4085 catalog_manager.cc:670] Loaded metadata for table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]
I20250109 20:55:35.558538 4085 catalog_manager.cc:670] Loaded metadata for table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]
I20250109 20:55:35.560292 4085 tablet_loader.cc:96] loaded metadata for tablet 07334099f3184a179000213a1839dddf (table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702])
I20250109 20:55:35.560979 4085 tablet_loader.cc:96] loaded metadata for tablet 0f95a0f62d3f4666adee4d1a51682f20 (table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31])
I20250109 20:55:35.561501 4085 tablet_loader.cc:96] loaded metadata for tablet 1010e1aeb5d140bf9f29d386bc195ed7 (table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc])
I20250109 20:55:35.562109 4085 tablet_loader.cc:96] loaded metadata for tablet 7df6e6361c074dc3984dbe5d92b1205d (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec])
I20250109 20:55:35.562443 4085 tablet_loader.cc:96] loaded metadata for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 (table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3])
I20250109 20:55:35.562763 4085 tablet_loader.cc:96] loaded metadata for tablet 8a47d3e2c8ea4690bbdbe87936af9415 (table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc])
I20250109 20:55:35.563231 4085 tablet_loader.cc:96] loaded metadata for tablet b685d1fbcd744586b8e0a3024e27a4ff (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c])
I20250109 20:55:35.563498 4085 tablet_loader.cc:96] loaded metadata for tablet c17492c45ef04ace967a0cd0df3a8eaf (table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869])
I20250109 20:55:35.563746 4085 tablet_loader.cc:96] loaded metadata for tablet c5339f2d6a1547c3aad25d50455eeab2 (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06])
I20250109 20:55:35.564531 4085 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:35.565624 4085 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:35.565719 4085 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:35.567265 4085 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:35.568303 4085 catalog_manager.cc:5910] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Loaded TSK: 0
I20250109 20:55:35.569342 4085 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:35.570375 4085 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:35.571297 4085 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:35.582370 3739 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37312:
name: "default.Table_0d2dc89f52bc4afc8d3de54485a3de62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:35.582904 3737 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37312:
name: "default.Table_2132f850190647e6af0ace1b48daa3ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:35.583700 3735 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37312:
table { table_name: "default.Table_94dbbe2d395441859b039d900093b7a2" } modify_external_catalogs: true
I20250109 20:55:35.584033 3735 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37312:
table { table_name: "default.Table_94dbbe2d395441859b039d900093b7a2" } modify_external_catalogs: true
I20250109 20:55:35.583943 3736 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37312:
table { table_name: "default.Table_cbd0b67985f24940af88114907c635eb" } new_table_name: "default.Table_61a48f808b814c4aadbc420991d9db06" modify_external_catalogs: true
I20250109 20:55:35.582437 3740 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37312:
name: "default.Table_22c1f44dd42c4195a778637ee4b876ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:35.585132 3738 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37312:
table { table_name: "default.Table_f5f640c97c72444396c24fbb34b9ec18" } modify_external_catalogs: true
I20250109 20:55:35.585405 3738 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37312:
table { table_name: "default.Table_f5f640c97c72444396c24fbb34b9ec18" } modify_external_catalogs: true
I20250109 20:55:35.585354 3734 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37312:
table { table_name: "default.Table_e27d0bbbd6a445f099a3be8e238bf9c8" } new_table_name: "default.Table_c66ec467f4264b9bbd248d748ba125c0" modify_external_catalogs: true
I20250109 20:55:35.585445 3733 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37312:
table { table_name: "default.Table_320b149b7d80477bb465bb9b3351d0be" } new_table_name: "default.Table_9c2e38f90afc4b9dbf223d20ffa931fb" modify_external_catalogs: true
I20250109 20:55:35.583349 3732 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37312:
name: "default.Table_66e7788f7db54a2ea368c19b84195df5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:35.584714 3731 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37312:
name: "default.Table_e672267cd18e4969b28037e40d7bc5cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250109 20:55:35.588835 3793 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cbd0b67985f24940af88114907c635eb
*** Check failure stack trace: ***
@ 0x7fabc00a2dcd google::LogMessage::Fail() at ??:0
@ 0x7fabc00a6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fabc00a27cc google::LogMessage::Flush() at ??:0
@ 0x7fabc00a2a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fabd62456de kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fabd62443c1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fabd624b4dc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fabd624b359 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
@ 0x7fabc2e000be kudu::Thread::SuperviseThread() at ??:0
@ 0x7fabc61e36db start_thread at ??:0
@ 0x7fabbef5371f clone at ??:0
W20250109 20:55:35.808763 3249 connection.cc:537] client connection to 127.2.96.125:37271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:35.808915 2970 connection.cc:537] client connection to 127.2.96.125:37271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:35.808902 3114 connection.cc:537] client connection to 127.2.96.125:37271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:35.810084 3368 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.96.125:37271)
W20250109 20:55:35.810802 3368 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.96.125:37271)
W20250109 20:55:35.811040 3368 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.125:37271)
W20250109 20:55:35.811355 3368 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.96.125:37271)
I20250109 20:55:36.182868 4102 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:36.183089 4102 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:36.183874 4102 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:36.185958 4005 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:36.186126 4002 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:36.186285 4002 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:36.186628 4102 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250109 20:55:37.283710 4107 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:37.284003 4107 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:37.284713 4107 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:37.286985 4002 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:37.287160 4005 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:37.287359 4005 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:37.287633 4107 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250109 20:55:37.716964 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--webserver_interface=127.2.96.126
--webserver_port=37329
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250109 20:55:38.064626 4112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:38.064947 4112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:38.065047 4112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:38.073030 4112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:38.073158 4112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:38.073204 4112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:38.073261 4112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:38.083097 4112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.96.126
--webserver_port=37329
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:38.084735 4112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:38.086566 4112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:38.098320 4117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:38.098902 4118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:38.099210 4120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:38.099880 4112 server_base.cc:1034] running on GCE node
I20250109 20:55:38.100725 4112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:38.102227 4112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:38.103451 4112 hybrid_clock.cc:648] HybridClock initialized: now 1736456138103360 us; error 70 us; skew 500 ppm
I20250109 20:55:38.103962 4112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:38.106438 4112 webserver.cc:458] Webserver started at http://127.2.96.126:37329/ using document root <none> and password file <none>
I20250109 20:55:38.107208 4112 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:38.107331 4112 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:38.112947 4112 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20250109 20:55:38.115958 4126 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:38.117280 4112 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250109 20:55:38.117424 4112 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
uuid: "b9908c8a5d72498890b9058df83a1b5e"
format_stamp: "Formatted at 2025-01-09 20:55:19 on dist-test-slave-thwl"
I20250109 20:55:38.118252 4112 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:38.130970 4112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:38.131825 4112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:38.132104 4112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:38.152853 4112 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.126:37541
I20250109 20:55:38.152871 4177 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.126:37541 every 8 connection(s)
I20250109 20:55:38.154482 4112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
I20250109 20:55:38.156594 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 4112
I20250109 20:55:38.162281 4178 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:55:38.165776 4178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap starting.
I20250109 20:55:38.181120 4178 log.cc:826] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:38.213878 4178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=29 ignored=0} mutations{seen=28 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:55:38.214519 4178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap complete.
I20250109 20:55:38.221072 4178 raft_consensus.cc:357] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:38.221951 4178 raft_consensus.cc:738] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Initialized, Role: FOLLOWER
I20250109 20:55:38.222759 4178 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 6.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:38.223693 4182 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:38.223980 4182 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:38.224758 4178 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:38.244661 4196 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:38.244895 4196 catalog_manager.cc:1553] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: loading cluster ID for follower catalog manager: success
I20250109 20:55:38.249608 4196 catalog_manager.cc:1575] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: acquiring CA information for follower catalog manager: success
I20250109 20:55:38.252867 4196 catalog_manager.cc:1603] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:38.253008 4112 master_runner.cc:186] Error getting master registration for 127.2.96.125:37271: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111),
I20250109 20:55:38.260818 4112 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:38.530880 3086 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:38.530973 3227 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:38.534125 4142 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cd7534821da847d0bf53f35562f4b653" instance_seqno: 1736456122119526) as {username='slave'} at 127.2.96.66:47489; Asking this server to re-register.
I20250109 20:55:38.534107 4143 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" instance_seqno: 1736456121613668) as {username='slave'} at 127.2.96.65:33469; Asking this server to re-register.
I20250109 20:55:38.535075 3086 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:38.535125 3227 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:38.537114 4143 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:38.537220 4142 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:38.649047 4202 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:38.649250 4202 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:38.649976 4202 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:38.653074 4002 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:38.658469 3362 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:38.658565 4153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 8 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "b9908c8a5d72498890b9058df83a1b5e" is_pre_election: true
I20250109 20:55:38.659291 4153 raft_consensus.cc:2405] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 95304c9e18024004b2277a53318ad77e for term 8 because replica has last-logged OpId of term: 6 index: 44, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 36.
I20250109 20:55:38.659915 4143 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" instance_seqno: 1736456122600579) as {username='slave'} at 127.2.96.67:41585; Asking this server to re-register.
I20250109 20:55:38.659934 4005 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:38.660310 4202 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250109 20:55:38.660454 3362 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:38.661685 4143 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:38.774824 4206 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:38.775091 4206 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:38.776762 4206 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:38.778352 4127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:38.779929 4127 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:38.783385 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 7 candidate_status { last_received { term: 6 index: 44 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e" is_pre_election: true
I20250109 20:55:38.783661 4028 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9908c8a5d72498890b9058df83a1b5e in current term 7: Already voted for candidate d84981b978e3495fb36dc33f09cc825a in this term.
I20250109 20:55:38.784224 4130 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e; no voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:38.784587 4206 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 6 FOLLOWER]: Advancing to term 7
I20250109 20:55:38.788333 4206 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250109 20:55:39.442337 4215 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:39.442533 4215 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:39.443202 4215 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:39.443758 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 8 candidate_status { last_received { term: 6 index: 44 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e" is_pre_election: true
I20250109 20:55:39.443990 4028 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 7.
I20250109 20:55:39.444454 4130 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e; no voters:
I20250109 20:55:39.444728 4215 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250109 20:55:39.444859 4215 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:39.444953 4215 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 7 FOLLOWER]: Advancing to term 8
W20250109 20:55:39.445259 4127 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:39.447160 4215 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:39.447652 4215 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 8 election: Requested vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:39.448166 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 8 candidate_status { last_received { term: 6 index: 44 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e"
I20250109 20:55:39.448413 4028 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 7 FOLLOWER]: Advancing to term 8
W20250109 20:55:39.448774 4127 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:39.450372 4028 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 8.
I20250109 20:55:39.450866 4130 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e; no voters: d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:39.451175 4215 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 8 FOLLOWER]: Leader election won for term 8
I20250109 20:55:39.451586 4215 raft_consensus.cc:695] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 8 LEADER]: Becoming Leader. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Running, Role: LEADER
I20250109 20:55:39.452047 4215 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 6.44, Last appended by leader: 44, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:39.454715 4218 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 8 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:39.455013 4218 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:39.455566 4220 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:39.458019 4220 catalog_manager.cc:670] Loaded metadata for table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]
I20250109 20:55:39.458477 4220 catalog_manager.cc:670] Loaded metadata for table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]
I20250109 20:55:39.458881 4220 catalog_manager.cc:670] Loaded metadata for table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]
I20250109 20:55:39.459229 4220 catalog_manager.cc:670] Loaded metadata for table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]
I20250109 20:55:39.459556 4220 catalog_manager.cc:670] Loaded metadata for table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]
I20250109 20:55:39.459836 4220 catalog_manager.cc:670] Loaded metadata for table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]
I20250109 20:55:39.460172 4220 catalog_manager.cc:670] Loaded metadata for table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]
I20250109 20:55:39.460500 4220 catalog_manager.cc:670] Loaded metadata for table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]
I20250109 20:55:39.460814 4220 catalog_manager.cc:670] Loaded metadata for table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]
I20250109 20:55:39.462313 4220 tablet_loader.cc:96] loaded metadata for tablet 07334099f3184a179000213a1839dddf (table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702])
I20250109 20:55:39.462917 4220 tablet_loader.cc:96] loaded metadata for tablet 0f95a0f62d3f4666adee4d1a51682f20 (table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31])
I20250109 20:55:39.463488 4220 tablet_loader.cc:96] loaded metadata for tablet 1010e1aeb5d140bf9f29d386bc195ed7 (table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc])
I20250109 20:55:39.464015 4220 tablet_loader.cc:96] loaded metadata for tablet 7df6e6361c074dc3984dbe5d92b1205d (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec])
I20250109 20:55:39.464267 4220 tablet_loader.cc:96] loaded metadata for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 (table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3])
I20250109 20:55:39.464527 4220 tablet_loader.cc:96] loaded metadata for tablet 8a47d3e2c8ea4690bbdbe87936af9415 (table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc])
I20250109 20:55:39.464948 4220 tablet_loader.cc:96] loaded metadata for tablet b685d1fbcd744586b8e0a3024e27a4ff (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c])
I20250109 20:55:39.465147 4220 tablet_loader.cc:96] loaded metadata for tablet c17492c45ef04ace967a0cd0df3a8eaf (table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869])
I20250109 20:55:39.465330 4220 tablet_loader.cc:96] loaded metadata for tablet c5339f2d6a1547c3aad25d50455eeab2 (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06])
I20250109 20:55:39.465999 4220 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:39.467027 4220 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:39.467096 4220 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:39.468297 4220 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:39.469147 4220 catalog_manager.cc:5910] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Loaded TSK: 0
I20250109 20:55:39.469980 4220 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:39.470896 4220 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:39.471659 4220 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:39.503698 4143 master_service.cc:946] ReplaceTablet: received request to replace tablet d8d7677ecbb9460192ca4323d38b03ad from {username='slave'} at 127.0.0.1:58500
I20250109 20:55:39.503831 4142 master_service.cc:946] ReplaceTablet: received request to replace tablet d8d7677ecbb9460192ca4323d38b03ad from {username='slave'} at 127.0.0.1:58500
I20250109 20:55:39.504038 4141 master_service.cc:946] ReplaceTablet: received request to replace tablet d8d7677ecbb9460192ca4323d38b03ad from {username='slave'} at 127.0.0.1:58500
I20250109 20:55:39.504653 4140 master_service.cc:946] ReplaceTablet: received request to replace tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 from {username='slave'} at 127.0.0.1:58500
I20250109 20:55:39.504829 4139 master_service.cc:946] ReplaceTablet: received request to replace tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 from {username='slave'} at 127.0.0.1:58500
I20250109 20:55:39.505568 4143 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58500:
table { table_name: "default.Table_edccb187ef7149b1a1d158edc0924690" } new_table_name: "default.Table_1b53d8836f984435902fbc1c6a9073b1" modify_external_catalogs: true
I20250109 20:55:39.505913 4138 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58500:
name: "default.Table_20e2171c365c4c3e8aafa501e51dae5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:39.505910 4142 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58500:
name: "default.Table_e992250ce97b4de3ac9cf07ff61775a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:39.507092 4141 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58500:
table { table_name: "default.Table_94dbbe2d395441859b039d900093b7a2" } modify_external_catalogs: true
I20250109 20:55:39.508479 4141 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58500:
table { table_name: "default.Table_94dbbe2d395441859b039d900093b7a2" } modify_external_catalogs: true
I20250109 20:55:39.508805 4134 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58500:
name: "default.Table_22c1f44dd42c4195a778637ee4b876ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:39.508270 4135 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58500:
table { table_name: "default.Table_f5f640c97c72444396c24fbb34b9ec18" } modify_external_catalogs: true
I20250109 20:55:39.509408 4135 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58500:
table { table_name: "default.Table_f5f640c97c72444396c24fbb34b9ec18" } modify_external_catalogs: true
I20250109 20:55:39.508446 4136 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58500:
name: "default.Table_2132f850190647e6af0ace1b48daa3ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:39.508244 4137 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58500:
table { table_name: "default.Table_cbd0b67985f24940af88114907c635eb" } new_table_name: "default.Table_61a48f808b814c4aadbc420991d9db06" modify_external_catalogs: true
I20250109 20:55:39.512653 4028 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Refusing update from remote peer b9908c8a5d72498890b9058df83a1b5e: Log matching property violated. Preceding OpId in replica: term: 3 index: 36. Preceding OpId from leader: term: 8 index: 46. (index mismatch)
I20250109 20:55:39.513468 4215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 36, Time since last communication: 0.000s
W20250109 20:55:39.514586 4127 consensus_peers.cc:487] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e -> Peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Couldn't send request to peer d84981b978e3495fb36dc33f09cc825a. Status: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250109 20:55:39.515995 4195 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cbd0b67985f24940af88114907c635eb
*** Check failure stack trace: ***
@ 0x7f2d2a28bdcd google::LogMessage::Fail() at ??:0
@ 0x7f2d2a28fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f2d2a28b7cc google::LogMessage::Flush() at ??:0
@ 0x7f2d2a28ba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f2d4042e6de kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250109 20:55:39.533095 4232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:39.533432 4232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:39.533931 4226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:39.534266 4226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:39.534207 4222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 8 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:39.534524 4222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:39.533910 4231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:39.534835 4231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f2d4042d3c1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250109 20:55:39.536374 4221 mvcc.cc:204] Tried to move back new op lower bound from 7112524347432599552 to 7112524347198849024. Current Snapshot: MvccSnapshot[applied={T|T < 7112524347432599552}]
I20250109 20:55:39.537593 4231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:39.537830 4231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f2d404344dc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250109 20:55:39.542594 4140 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Sending DeleteTablet for 3 replicas of tablet 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:55:39.544663 4140 catalog_manager.cc:6596] ReplaceTablet: tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 deleted and replaced by tablet 83ac24dec39543e7a617bf4fc3366c10
@ 0x7f2d40434359 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
I20250109 20:55:39.552820 3161 tablet_service.cc:1514] Processing DeleteTablet for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 with delete_type TABLET_DATA_DELETED (replaced by tablet 83ac24dec39543e7a617bf4fc3366c10) from {username='slave'} at 127.0.0.1:53340
@ 0x7f2d2cfe90be kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2d303cc6db start_thread at ??:0
I20250109 20:55:39.564523 3019 tablet_service.cc:1514] Processing DeleteTablet for tablet 84eb8b3ef2b0414da0e9de1ba8c4a129 with delete_type TABLET_DATA_DELETED (replaced by tablet 83ac24dec39543e7a617bf4fc3366c10) from {username='slave'} at 127.0.0.1:49392
@ 0x7f2d2913c71f clone at ??:0
I20250109 20:55:39.567428 4242 tablet_replica.cc:331] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: stopping tablet replica
I20250109 20:55:39.567649 4242 raft_consensus.cc:2238] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:55:39.567762 4243 tablet_replica.cc:331] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: stopping tablet replica
I20250109 20:55:39.567901 4242 raft_consensus.cc:2267] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:39.567939 4243 raft_consensus.cc:2238] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:55:39.568325 4243 raft_consensus.cc:2267] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:39.568571 4242 ts_tablet_manager.cc:1905] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:39.568871 4243 ts_tablet_manager.cc:1905] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:39.575176 4242 ts_tablet_manager.cc:1918] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:55:39.575176 4243 ts_tablet_manager.cc:1918] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:55:39.575294 4242 log.cc:1198] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-1/wal/wals/84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:55:39.575294 4243 log.cc:1198] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/wals/84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:55:39.575726 4243 ts_tablet_manager.cc:1939] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0d843776b4dc4cc194def9bd50945c29: Deleting consensus metadata
I20250109 20:55:39.575747 4242 ts_tablet_manager.cc:1939] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P cd7534821da847d0bf53f35562f4b653: Deleting consensus metadata
W20250109 20:55:39.731009 3115 connection.cc:537] server connection from 127.0.0.1:53340 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:39.731040 2973 connection.cc:537] server connection from 127.0.0.1:49392 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:39.731499 4241 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:55:39.551459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.2.96.67:41259)
0109 20:55:39.566771 (+ 15312us) server_negotiation.cc:206] Beginning negotiation
0109 20:55:39.566778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 20:55:39.566798 (+ 20us) server_negotiation.cc:407] Connection header received
0109 20:55:39.566870 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 20:55:39.566877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 20:55:39.566981 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 20:55:39.567159 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 20:55:39.731171 (+164012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41588: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":15168,"thread_start_us":16113,"threads_started":1}
W20250109 20:55:39.731686 3371 connection.cc:537] client connection to 127.2.96.126:37541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:39.732278 3252 connection.cc:537] client connection to 127.2.96.126:37541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:39.732717 3371 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.96.126:37541)
W20250109 20:55:39.733194 3371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:39.733481 3371 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:39.733716 3371 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.96.126:37541)
I20250109 20:55:39.898095 3982 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:55:39.898257 3982 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:40.037672 4231 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:40.037885 4231 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:40.038594 4231 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:40.040382 4005 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:40.040563 4002 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:40.040675 4002 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:40.040906 4231 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250109 20:55:40.734985 4258 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:40.735210 4258 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:40.735802 4258 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:40.736573 4005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250109 20:55:40.737586 4002 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:40.737783 4005 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:40.737911 4005 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:40.738160 4258 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250109 20:55:40.973927 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.125:37271
--webserver_interface=127.2.96.125
--webserver_port=39189
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250109 20:55:41.297081 4262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:41.297371 4262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:41.297485 4262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:41.305131 4262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:41.305231 4262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:41.305289 4262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:41.305367 4262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:41.314797 4262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.125:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.96.125
--webserver_port=39189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:41.316409 4262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:41.318555 4262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 20:55:41.326959 4271 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:41.327155 4271 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:41.327888 4271 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:41.329763 4268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:41.330129 4005 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:41.330168 4267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:41.331339 4270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:41.331400 4002 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:41.331547 4002 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:41.331807 4271 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250109 20:55:41.332911 4262 server_base.cc:1034] running on GCE node
I20250109 20:55:41.333694 4262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:41.335206 4262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:41.336444 4262 hybrid_clock.cc:648] HybridClock initialized: now 1736456141336375 us; error 45 us; skew 500 ppm
I20250109 20:55:41.336804 4262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:41.339304 4262 webserver.cc:458] Webserver started at http://127.2.96.125:39189/ using document root <none> and password file <none>
I20250109 20:55:41.340027 4262 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:41.340137 4262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:41.345405 4262 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250109 20:55:41.348168 4280 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:41.349468 4262 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250109 20:55:41.349617 4262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
uuid: "d84981b978e3495fb36dc33f09cc825a"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:41.350553 4262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:41.366914 4262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:41.367884 4262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:41.368145 4262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:41.386596 4262 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.125:37271
I20250109 20:55:41.386631 4332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.125:37271 every 8 connection(s)
I20250109 20:55:41.388024 4262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
I20250109 20:55:41.392575 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 4262
I20250109 20:55:41.395609 4333 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:55:41.399243 4333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap starting.
I20250109 20:55:41.411872 4333 log.cc:826] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:41.446041 4333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=44 ignored=0} inserts{seen=29 ignored=0} mutations{seen=28 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250109 20:55:41.446745 4333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap complete.
I20250109 20:55:41.453053 4333 raft_consensus.cc:357] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 7 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:41.454013 4333 raft_consensus.cc:738] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84981b978e3495fb36dc33f09cc825a, State: Initialized, Role: FOLLOWER
I20250109 20:55:41.454656 4333 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 7.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:41.455438 4337 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:41.455754 4337 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:41.456478 4333 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:41.469686 4351 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:41.469921 4351 catalog_manager.cc:1553] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: loading cluster ID for follower catalog manager: success
I20250109 20:55:41.475522 4351 catalog_manager.cc:1575] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: acquiring CA information for follower catalog manager: success
I20250109 20:55:41.479440 4351 catalog_manager.cc:1603] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:41.479851 4262 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:55:41.485507 4262 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:41.835152 3363 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:41.838688 4298 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" instance_seqno: 1736456122600579) as {username='slave'} at 127.2.96.67:52679; Asking this server to re-register.
I20250109 20:55:41.840194 3363 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:41.841519 3228 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:41.841753 3087 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:41.842428 4298 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:41.843201 4297 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" instance_seqno: 1736456121613668) as {username='slave'} at 127.2.96.65:51785; Asking this server to re-register.
I20250109 20:55:41.843278 4296 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cd7534821da847d0bf53f35562f4b653" instance_seqno: 1736456122119526) as {username='slave'} at 127.2.96.66:41381; Asking this server to re-register.
I20250109 20:55:41.843726 3087 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:41.843874 3228 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:41.845057 4296 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:41.845232 4297 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:41.923063 4337 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:41.923345 4337 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:41.925099 4337 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:41.925835 4284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:41.927691 4284 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:41.931591 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 8 candidate_status { last_received { term: 7 index: 45 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e" is_pre_election: true
I20250109 20:55:41.931856 4028 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d84981b978e3495fb36dc33f09cc825a in current term 8: Already voted for candidate b9908c8a5d72498890b9058df83a1b5e in this term.
I20250109 20:55:41.932502 4284 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:41.932868 4337 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 7 FOLLOWER]: Advancing to term 8
I20250109 20:55:41.936658 4337 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250109 20:55:41.939436 4360 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:41.939596 4360 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:41.940184 4360 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:41.942129 4005 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:41.946240 4308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
I20250109 20:55:41.946678 4308 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95304c9e18024004b2277a53318ad77e in term 8.
I20250109 20:55:41.947088 4002 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a; no voters: b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:41.947362 4360 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250109 20:55:41.947460 4360 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:41.947553 4360 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 8 FOLLOWER]: Advancing to term 9
I20250109 20:55:41.949450 4360 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:41.949957 4360 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 election: Requested vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
I20250109 20:55:41.950429 4308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95304c9e18024004b2277a53318ad77e" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a"
I20250109 20:55:41.950606 4308 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 8 FOLLOWER]: Advancing to term 9
W20250109 20:55:41.951107 4005 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:41.952849 4308 raft_consensus.cc:2463] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95304c9e18024004b2277a53318ad77e in term 9.
I20250109 20:55:41.953209 4002 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a; no voters: b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:41.953447 4360 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 9 FOLLOWER]: Leader election won for term 9
I20250109 20:55:41.953817 4360 raft_consensus.cc:695] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 9 LEADER]: Becoming Leader. State: Replica: 95304c9e18024004b2277a53318ad77e, State: Running, Role: LEADER
I20250109 20:55:41.954219 4360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 8.46, Last appended by leader: 46, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:41.955839 4364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95304c9e18024004b2277a53318ad77e. Latest consensus state: current_term: 9 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:41.956132 4364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:41.956779 4366 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:41.959481 4366 catalog_manager.cc:670] Loaded metadata for table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]
I20250109 20:55:41.959975 4366 catalog_manager.cc:670] Loaded metadata for table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]
I20250109 20:55:41.960368 4366 catalog_manager.cc:670] Loaded metadata for table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]
I20250109 20:55:41.960791 4366 catalog_manager.cc:670] Loaded metadata for table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]
I20250109 20:55:41.961153 4366 catalog_manager.cc:670] Loaded metadata for table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]
I20250109 20:55:41.961444 4366 catalog_manager.cc:670] Loaded metadata for table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]
I20250109 20:55:41.961719 4366 catalog_manager.cc:670] Loaded metadata for table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]
I20250109 20:55:41.962002 4366 catalog_manager.cc:670] Loaded metadata for table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]
I20250109 20:55:41.962281 4366 catalog_manager.cc:670] Loaded metadata for table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]
I20250109 20:55:41.963797 4366 tablet_loader.cc:96] loaded metadata for tablet 07334099f3184a179000213a1839dddf (table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702])
I20250109 20:55:41.964321 4366 tablet_loader.cc:96] loaded metadata for tablet 0f95a0f62d3f4666adee4d1a51682f20 (table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31])
I20250109 20:55:41.964736 4366 tablet_loader.cc:96] loaded metadata for tablet 1010e1aeb5d140bf9f29d386bc195ed7 (table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc])
I20250109 20:55:41.965252 4366 tablet_loader.cc:96] loaded metadata for tablet 7df6e6361c074dc3984dbe5d92b1205d (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec])
I20250109 20:55:41.965487 4366 tablet_loader.cc:96] loaded metadata for tablet 83ac24dec39543e7a617bf4fc3366c10 (table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3])
I20250109 20:55:41.965885 4366 tablet_loader.cc:96] loaded metadata for tablet 8a47d3e2c8ea4690bbdbe87936af9415 (table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc])
I20250109 20:55:41.966280 4366 tablet_loader.cc:96] loaded metadata for tablet b685d1fbcd744586b8e0a3024e27a4ff (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c])
I20250109 20:55:41.966553 4366 tablet_loader.cc:96] loaded metadata for tablet c17492c45ef04ace967a0cd0df3a8eaf (table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869])
I20250109 20:55:41.966812 4366 tablet_loader.cc:96] loaded metadata for tablet c5339f2d6a1547c3aad25d50455eeab2 (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06])
I20250109 20:55:41.967571 4366 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:41.968480 4366 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:41.968549 4366 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:41.969810 4366 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:41.970762 4366 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Loaded TSK: 0
I20250109 20:55:41.971613 4366 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:41.972445 4366 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:41.973242 4366 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:42.008317 4017 master_service.cc:946] ReplaceTablet: received request to replace tablet dc461d653b944163bcf564e04efb74e2 from {username='slave'} at 127.0.0.1:48604
I20250109 20:55:42.008456 4016 master_service.cc:946] ReplaceTablet: received request to replace tablet 8a47d3e2c8ea4690bbdbe87936af9415 from {username='slave'} at 127.0.0.1:48604
I20250109 20:55:42.008824 4018 master_service.cc:946] ReplaceTablet: received request to replace tablet b685d1fbcd744586b8e0a3024e27a4ff from {username='slave'} at 127.0.0.1:48604
I20250109 20:55:42.009676 4012 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48604:
name: "default.Table_5dcc9f9689084615b4c47a34fdd1a329"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:42.010259 4010 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48604:
table { table_name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2" } modify_external_catalogs: true
I20250109 20:55:42.010612 4010 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48604:
table { table_name: "default.Table_da5d7a22c7704d4facb0ae576942a0d2" } modify_external_catalogs: true
I20250109 20:55:42.010977 4014 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48604:
name: "default.Table_aabd822000834e5b9e47ffbb942b54ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:42.011337 4011 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48604:
table { table_name: "default.Table_84db4e769e42410f985d1d5f5cbe527a" } new_table_name: "default.Table_a2481e08ab1c4f2193462e658e9dbecc" modify_external_catalogs: true
I20250109 20:55:42.011685 4015 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48604:
name: "default.Table_48885eaf759344b8a2b07e6b94fa13ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:42.011904 4017 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48604:
name: "default.Table_66e7788f7db54a2ea368c19b84195df5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:42.010623 4013 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48604:
table { table_name: "default.Table_d2f3663d9ed949609d925f9bb37f7633" } modify_external_catalogs: true
I20250109 20:55:42.012442 4013 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48604:
table { table_name: "default.Table_d2f3663d9ed949609d925f9bb37f7633" } modify_external_catalogs: true
I20250109 20:55:42.009725 4009 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48604:
name: "default.Table_e672267cd18e4969b28037e40d7bc5cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:42.016023 4308 raft_consensus.cc:1270] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Refusing update from remote peer 95304c9e18024004b2277a53318ad77e: Log matching property violated. Preceding OpId in replica: term: 7 index: 45. Preceding OpId from leader: term: 9 index: 48. (index mismatch)
I20250109 20:55:42.016860 4364 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 44, Time since last communication: 0.000s
I20250109 20:55:42.017251 4364 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [LEADER]: Peer d84981b978e3495fb36dc33f09cc825a log is divergent from this leader: its last log entry 7.45 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 44
W20250109 20:55:42.017812 4005 consensus_peers.cc:487] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e -> Peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Couldn't send request to peer b9908c8a5d72498890b9058df83a1b5e. Status: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 20:55:42.018934 4308 pending_rounds.cc:77] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Aborting all ops after (but not including) 44
I20250109 20:55:42.019081 4308 pending_rounds.cc:99] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Aborting uncommitted NO_OP operation due to leader change: 7.45
I20250109 20:55:42.019237 4308 raft_consensus.cc:2883] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
F20250109 20:55:42.019600 4070 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cbd0b67985f24940af88114907c635eb
*** Check failure stack trace: ***
@ 0x7fcac9698dcd google::LogMessage::Fail() at ??:0
@ 0x7fcac969cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fcac96987cc google::LogMessage::Flush() at ??:0
@ 0x7fcac9698a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fcadf83b6de kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fcadf83a3c1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fcadf8414dc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250109 20:55:42.044462 4337 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95304c9e18024004b2277a53318ad77e. Latest consensus state: current_term: 9 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:42.044804 4337 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:42.044734 4372 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:42.045042 4372 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fcadf841359 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
@ 0x7fcacc3f60be kudu::Thread::SuperviseThread() at ??:0
I20250109 20:55:42.054364 4372 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "95304c9e18024004b2277a53318ad77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:42.054615 4372 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:42.055851 4016 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Sending DeleteTablet for 3 replicas of tablet 8a47d3e2c8ea4690bbdbe87936af9415
I20250109 20:55:42.057978 4016 catalog_manager.cc:6596] ReplaceTablet: tablet 8a47d3e2c8ea4690bbdbe87936af9415 deleted and replaced by tablet 2b5cf5740eb642f6aae83df3916cbf3f
@ 0x7fcacf7d96db start_thread at ??:0
@ 0x7fcac854971f clone at ??:0
W20250109 20:55:42.234150 4382 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:55:42.058810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.2.96.67:41259)
0109 20:55:42.063252 (+ 4442us) server_negotiation.cc:206] Beginning negotiation
0109 20:55:42.063259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 20:55:42.233841 (+170582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41600: BlockingRecv error: recv got EOF from 127.0.0.1:41600 (error 108)
Metrics: {"server-negotiator.queue_time_us":4304,"thread_start_us":4187,"threads_started":1}
W20250109 20:55:42.234290 3371 connection.cc:537] client connection to 127.2.96.124:46741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:42.234741 3252 connection.cc:537] client connection to 127.2.96.124:46741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:42.235015 3371 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.96.124:46741)
W20250109 20:55:42.235467 3371 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.96.124:46741)
W20250109 20:55:42.235463 4379 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:55:42.058084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.2.96.65:33615)
0109 20:55:42.063116 (+ 5032us) server_negotiation.cc:206] Beginning negotiation
0109 20:55:42.063135 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0109 20:55:42.063162 (+ 27us) server_negotiation.cc:407] Connection header received
0109 20:55:42.063257 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 20:55:42.063264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 20:55:42.063366 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 20:55:42.063480 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 20:55:42.233851 (+170371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49396: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":529,"threads_started":1}
W20250109 20:55:42.235734 3371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.124:46741)
W20250109 20:55:42.235986 3371 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.96.124:46741)
W20250109 20:55:42.236579 4380 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:55:42.058295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.2.96.66:35185)
0109 20:55:42.063654 (+ 5359us) server_negotiation.cc:206] Beginning negotiation
0109 20:55:42.063663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0109 20:55:42.063683 (+ 20us) server_negotiation.cc:407] Connection header received
0109 20:55:42.063749 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 20:55:42.063754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 20:55:42.063835 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 20:55:42.063939 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 20:55:42.234766 (+170827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53356: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":5226,"thread_start_us":5112,"threads_started":1}
I20250109 20:55:42.452913 4391 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:55:42.453110 4391 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:42.453895 4391 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:55:42.454949 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:55:42.455189 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:55:42.455332 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:55:42.455611 4391 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:55:42.547657 4372 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 95304c9e18024004b2277a53318ad77e)
I20250109 20:55:42.547901 4372 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:42.548614 4372 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:42.550849 4284 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:42.551064 4284 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:42.551146 4284 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:42.551388 4372 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250109 20:55:43.103125 4396 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 95304c9e18024004b2277a53318ad77e)
I20250109 20:55:43.103371 4396 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:43.104071 4396 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:43.106822 4284 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:43.107048 4284 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:43.107179 4284 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:43.107440 4396 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250109 20:55:43.876572 2433 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskljSzmY/build/asan/bin/kudu pid 4262
I20250109 20:55:43.888468 4328 generic_service.cc:196] Checking for leaks (request via RPC)
I20250109 20:55:43.993292 4404 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 95304c9e18024004b2277a53318ad77e)
I20250109 20:55:43.993525 2433 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskljSzmY/build/asan/bin/kudu with pid 4262
I20250109 20:55:44.771870 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.125:37271
--webserver_interface=127.2.96.125
--webserver_port=39189
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250109 20:55:45.097254 4406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:45.097579 4406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:45.097702 4406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:45.105747 4406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:45.105858 4406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:45.105904 4406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:45.105962 4406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:45.116626 4406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.125:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.96.125
--webserver_port=39189
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:45.118206 4406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:45.119920 4406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:45.130187 4412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:45.130925 4413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:45.131582 4415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:45.132246 4406 server_base.cc:1034] running on GCE node
I20250109 20:55:45.133184 4406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:45.134886 4406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:45.136119 4406 hybrid_clock.cc:648] HybridClock initialized: now 1736456145136062 us; error 32 us; skew 500 ppm
I20250109 20:55:45.136497 4406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:45.138990 4406 webserver.cc:458] Webserver started at http://127.2.96.125:39189/ using document root <none> and password file <none>
I20250109 20:55:45.139930 4406 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:45.140058 4406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:45.146430 4406 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20250109 20:55:45.149150 4421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:45.150571 4406 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250109 20:55:45.150725 4406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
uuid: "d84981b978e3495fb36dc33f09cc825a"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:45.151592 4406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:45.168452 4406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:45.169306 4406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:45.169621 4406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:45.187166 4406 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.125:37271
I20250109 20:55:45.187175 4472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.125:37271 every 8 connection(s)
I20250109 20:55:45.188591 4406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-1/data/info.pb
I20250109 20:55:45.189903 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 4406
I20250109 20:55:45.195524 4473 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:55:45.199976 4473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap starting.
I20250109 20:55:45.212381 4473 log.cc:826] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:45.245335 4473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap replayed 1/1 log segments. Stats: ops{read=50 overwritten=1 applied=49 ignored=0} inserts{seen=32 ignored=0} mutations{seen=31 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:55:45.245903 4473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Bootstrap complete.
I20250109 20:55:45.252116 4473 raft_consensus.cc:357] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:45.252872 4473 raft_consensus.cc:738] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84981b978e3495fb36dc33f09cc825a, State: Initialized, Role: FOLLOWER
I20250109 20:55:45.253486 4473 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 9.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:45.254257 4478 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:45.254545 4478 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:45.255389 4473 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:45.267688 4492 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:45.267863 4492 catalog_manager.cc:1553] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: loading cluster ID for follower catalog manager: success
I20250109 20:55:45.272461 4492 catalog_manager.cc:1575] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: acquiring CA information for follower catalog manager: success
I20250109 20:55:45.265794 4406 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:55:45.273819 4406 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:45.275322 4492 catalog_manager.cc:1603] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:45.365293 3228 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:45.365339 3363 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:45.368409 4438 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" instance_seqno: 1736456122600579) as {username='slave'} at 127.2.96.67:42393; Asking this server to re-register.
I20250109 20:55:45.368435 4437 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cd7534821da847d0bf53f35562f4b653" instance_seqno: 1736456122119526) as {username='slave'} at 127.2.96.66:36327; Asking this server to re-register.
I20250109 20:55:45.369136 3363 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:45.369248 3087 heartbeater.cc:346] Connected to a master server at 127.2.96.125:37271
I20250109 20:55:45.369665 3228 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:45.370784 4437 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" instance_seqno: 1736456121613668) as {username='slave'} at 127.2.96.65:47495; Asking this server to re-register.
I20250109 20:55:45.371412 4438 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:45.371531 3087 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:45.371587 4436 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:45.372973 4438 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:45.732537 4478 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:45.732800 4478 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:45.734598 4478 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:45.735399 4425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:45.736689 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:45.736907 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:45.736996 4425 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:45.737389 4478 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250109 20:55:46.259022 4500 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:46.259210 4500 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:46.259862 4500 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:46.262485 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:46.262655 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:46.262763 4425 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:46.263063 4500 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250109 20:55:46.808425 4504 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:55:46.808670 4504 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:46.809327 4504 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:55:46.810156 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:55:46.810307 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:55:46.810454 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:55:46.810678 4504 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:55:47.181093 4505 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:47.181277 4505 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:47.181845 4505 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:47.184037 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:47.184244 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:47.184340 4425 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:47.184598 4505 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250109 20:55:48.231390 4510 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:48.231599 4510 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:48.232219 4510 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:48.235374 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:48.235566 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:48.235659 4425 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e
I20250109 20:55:48.235890 4510 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250109 20:55:48.261106 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--webserver_interface=127.2.96.126
--webserver_port=37329
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250109 20:55:48.589685 4514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:48.590013 4514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:48.590140 4514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:48.597505 4514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:48.597643 4514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:48.597719 4514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:48.597795 4514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:48.606657 4514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.126:37541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.96.126
--webserver_port=37329
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:48.608223 4514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:48.609843 4514 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:48.621892 4520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:48.623170 4519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:48.623369 4522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:48.623907 4514 server_base.cc:1034] running on GCE node
I20250109 20:55:48.624825 4514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:48.626428 4514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:48.627648 4514 hybrid_clock.cc:648] HybridClock initialized: now 1736456148627583 us; error 43 us; skew 500 ppm
I20250109 20:55:48.627957 4514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:48.630331 4514 webserver.cc:458] Webserver started at http://127.2.96.126:37329/ using document root <none> and password file <none>
I20250109 20:55:48.631104 4514 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:48.631219 4514 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:48.636265 4514 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250109 20:55:48.638926 4528 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:48.640197 4514 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250109 20:55:48.640370 4514 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
uuid: "b9908c8a5d72498890b9058df83a1b5e"
format_stamp: "Formatted at 2025-01-09 20:55:19 on dist-test-slave-thwl"
I20250109 20:55:48.641223 4514 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:48.681519 4514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:48.682358 4514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:48.682646 4514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:48.699728 4514 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.126:37541
I20250109 20:55:48.699759 4579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.126:37541 every 8 connection(s)
I20250109 20:55:48.701207 4514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-0/data/info.pb
I20250109 20:55:48.707259 4580 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:55:48.710254 4580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap starting.
I20250109 20:55:48.710513 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 4514
I20250109 20:55:48.724560 4580 log.cc:826] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:48.758832 3362 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:48.760742 3086 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:48.764303 4580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=46 overwritten=0 applied=46 ignored=0} inserts{seen=30 ignored=0} mutations{seen=29 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:55:48.765053 4580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Bootstrap complete.
I20250109 20:55:48.765630 3227 heartbeater.cc:346] Connected to a master server at 127.2.96.126:37541
I20250109 20:55:48.771565 4580 raft_consensus.cc:357] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:48.772332 4580 raft_consensus.cc:738] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Initialized, Role: FOLLOWER
I20250109 20:55:48.772908 4580 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 46, Last appended: 8.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:48.773660 4586 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:48.773918 4586 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:48.774801 4580 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:48.789876 4600 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:48.790069 4600 catalog_manager.cc:1553] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: loading cluster ID for follower catalog manager: success
I20250109 20:55:48.794039 4600 catalog_manager.cc:1575] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: acquiring CA information for follower catalog manager: success
I20250109 20:55:48.796765 4600 catalog_manager.cc:1603] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:48.811954 4514 master_runner.cc:186] Error getting master registration for 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111),
I20250109 20:55:48.819386 4514 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:49.209340 4586 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:49.209609 4586 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:49.211444 4586 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:49.212126 4532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:49.212159 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
I20250109 20:55:49.212687 4448 raft_consensus.cc:2388] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9908c8a5d72498890b9058df83a1b5e in current term 9: Already voted for candidate 95304c9e18024004b2277a53318ad77e in this term.
W20250109 20:55:49.213482 4532 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:49.213611 4532 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e; no voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:49.213948 4586 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 8 FOLLOWER]: Advancing to term 9
I20250109 20:55:49.217783 4586 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250109 20:55:49.264430 4544 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" instance_seqno: 1736456122600579) as {username='slave'} at 127.2.96.67:59111; Asking this server to re-register.
I20250109 20:55:49.264474 4545 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" instance_seqno: 1736456121613668) as {username='slave'} at 127.2.96.65:55277; Asking this server to re-register.
I20250109 20:55:49.265144 3086 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:49.265206 3362 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:49.266698 4544 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:49.266731 4545 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:49.268036 4544 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cd7534821da847d0bf53f35562f4b653" instance_seqno: 1736456122119526) as {username='slave'} at 127.2.96.66:55795; Asking this server to re-register.
I20250109 20:55:49.268515 3227 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:49.269591 4544 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:49.741421 4609 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:49.741600 4609 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:49.742280 4609 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:49.742866 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 10 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "d84981b978e3495fb36dc33f09cc825a" is_pre_election: true
I20250109 20:55:49.743203 4448 raft_consensus.cc:2405] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9908c8a5d72498890b9058df83a1b5e for term 10 because replica has last-logged OpId of term: 9 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 46.
W20250109 20:55:49.744103 4532 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:49.744221 4532 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e; no voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:49.744475 4609 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250109 20:55:50.288697 4406 master_runner.cc:186] Error getting master registration for 127.2.96.124:46741: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111),
I20250109 20:55:50.288859 4406 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:50.499540 4622 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:55:50.499730 4622 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:50.500432 4622 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:50.501101 4555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 10 candidate_status { last_received { term: 9 index: 49 } } ignore_live_leader: false dest_uuid: "b9908c8a5d72498890b9058df83a1b5e" is_pre_election: true
I20250109 20:55:50.501574 4555 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d84981b978e3495fb36dc33f09cc825a in term 9.
I20250109 20:55:50.502334 4425 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters:
I20250109 20:55:50.502669 4622 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250109 20:55:50.502669 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:50.502843 4622 raft_consensus.cc:491] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:55:50.502947 4622 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 9 FOLLOWER]: Advancing to term 10
I20250109 20:55:50.506603 4622 raft_consensus.cc:513] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:50.507180 4622 leader_election.cc:290] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 election: Requested vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:50.507570 4555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d84981b978e3495fb36dc33f09cc825a" candidate_term: 10 candidate_status { last_received { term: 9 index: 49 } } ignore_live_leader: false dest_uuid: "b9908c8a5d72498890b9058df83a1b5e"
I20250109 20:55:50.507741 4555 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 9 FOLLOWER]: Advancing to term 10
W20250109 20:55:50.508488 4425 leader_election.cc:336] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:50.509827 4555 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d84981b978e3495fb36dc33f09cc825a in term 10.
I20250109 20:55:50.510214 4425 leader_election.cc:304] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a; no voters: 95304c9e18024004b2277a53318ad77e
I20250109 20:55:50.510500 4622 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 10 FOLLOWER]: Leader election won for term 10
I20250109 20:55:50.510921 4622 raft_consensus.cc:695] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [term 10 LEADER]: Becoming Leader. State: Replica: d84981b978e3495fb36dc33f09cc825a, State: Running, Role: LEADER
I20250109 20:55:50.511392 4622 consensus_queue.cc:237] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 9.49, Last appended by leader: 49, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:50.514045 4624 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84981b978e3495fb36dc33f09cc825a. Latest consensus state: current_term: 10 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:50.514346 4624 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:50.514946 4626 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:50.517388 4626 catalog_manager.cc:670] Loaded metadata for table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]
I20250109 20:55:50.517796 4626 catalog_manager.cc:670] Loaded metadata for table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]
I20250109 20:55:50.518112 4626 catalog_manager.cc:670] Loaded metadata for table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]
I20250109 20:55:50.518440 4626 catalog_manager.cc:670] Loaded metadata for table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]
I20250109 20:55:50.518772 4626 catalog_manager.cc:670] Loaded metadata for table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]
I20250109 20:55:50.519055 4626 catalog_manager.cc:670] Loaded metadata for table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]
I20250109 20:55:50.519336 4626 catalog_manager.cc:670] Loaded metadata for table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]
I20250109 20:55:50.519634 4626 catalog_manager.cc:670] Loaded metadata for table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]
I20250109 20:55:50.519901 4626 catalog_manager.cc:670] Loaded metadata for table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]
I20250109 20:55:50.521430 4626 tablet_loader.cc:96] loaded metadata for tablet 07334099f3184a179000213a1839dddf (table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702])
I20250109 20:55:50.521934 4626 tablet_loader.cc:96] loaded metadata for tablet 0f95a0f62d3f4666adee4d1a51682f20 (table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31])
I20250109 20:55:50.522331 4626 tablet_loader.cc:96] loaded metadata for tablet 1010e1aeb5d140bf9f29d386bc195ed7 (table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc])
I20250109 20:55:50.522569 4626 tablet_loader.cc:96] loaded metadata for tablet 2b5cf5740eb642f6aae83df3916cbf3f (table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc])
I20250109 20:55:50.522846 4626 tablet_loader.cc:96] loaded metadata for tablet 45574543f3dd492c852637bd9739fcf3 (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c])
I20250109 20:55:50.523391 4626 tablet_loader.cc:96] loaded metadata for tablet 7df6e6361c074dc3984dbe5d92b1205d (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec])
I20250109 20:55:50.523625 4626 tablet_loader.cc:96] loaded metadata for tablet 83ac24dec39543e7a617bf4fc3366c10 (table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3])
I20250109 20:55:50.524479 4626 tablet_loader.cc:96] loaded metadata for tablet c17492c45ef04ace967a0cd0df3a8eaf (table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869])
I20250109 20:55:50.524717 4626 tablet_loader.cc:96] loaded metadata for tablet c5339f2d6a1547c3aad25d50455eeab2 (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06])
I20250109 20:55:50.525422 4626 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:50.526422 4626 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:50.526502 4626 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:50.527819 4626 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:50.528826 4626 catalog_manager.cc:5910] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Loaded TSK: 0
I20250109 20:55:50.529718 4626 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:50.530687 4626 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:50.531549 4626 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:50.556581 4436 master_service.cc:946] ReplaceTablet: received request to replace tablet dc461d653b944163bcf564e04efb74e2 from {username='slave'} at 127.0.0.1:47370
I20250109 20:55:50.556779 4437 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f95a0f62d3f4666adee4d1a51682f20 from {username='slave'} at 127.0.0.1:47370
I20250109 20:55:50.557370 4438 master_service.cc:946] ReplaceTablet: received request to replace tablet 8a47d3e2c8ea4690bbdbe87936af9415 from {username='slave'} at 127.0.0.1:47370
I20250109 20:55:50.558772 4438 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47370:
name: "default.Table_aabd822000834e5b9e47ffbb942b54ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:50.559408 4429 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "default.Table_d2f3663d9ed949609d925f9bb37f7633" } modify_external_catalogs: true
I20250109 20:55:50.562247 4436 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47370:
name: "default.Table_48885eaf759344b8a2b07e6b94fa13ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:50.559796 4431 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "default.Table_f5f640c97c72444396c24fbb34b9ec18" } modify_external_catalogs: true
I20250109 20:55:50.559702 4432 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47370:
name: "default.Table_22c1f44dd42c4195a778637ee4b876ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:50.563426 4429 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "default.Table_d2f3663d9ed949609d925f9bb37f7633" } modify_external_catalogs: true
I20250109 20:55:50.563097 4431 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "default.Table_f5f640c97c72444396c24fbb34b9ec18" } modify_external_catalogs: true
I20250109 20:55:50.562469 4433 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47370:
name: "default.Table_0d2dc89f52bc4afc8d3de54485a3de62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:50.564113 4430 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47370:
name: "default.Table_2132f850190647e6af0ace1b48daa3ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:50.565671 4434 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "default.Table_e27d0bbbd6a445f099a3be8e238bf9c8" } new_table_name: "default.Table_c66ec467f4264b9bbd248d748ba125c0" modify_external_catalogs: true
I20250109 20:55:50.566011 4435 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "default.Table_320b149b7d80477bb465bb9b3351d0be" } new_table_name: "default.Table_9c2e38f90afc4b9dbf223d20ffa931fb" modify_external_catalogs: true
F20250109 20:55:50.568949 4491 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cbd0b67985f24940af88114907c635eb
*** Check failure stack trace: ***
I20250109 20:55:50.569569 4555 raft_consensus.cc:1270] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Refusing update from remote peer d84981b978e3495fb36dc33f09cc825a: Log matching property violated. Preceding OpId in replica: term: 8 index: 46. Preceding OpId from leader: term: 10 index: 51. (index mismatch)
I20250109 20:55:50.570554 4622 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 46, Time since last communication: 0.000s
@ 0x7f846f8afdcd google::LogMessage::Fail() at ??:0
W20250109 20:55:50.571633 4425 consensus_peers.cc:487] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a -> Peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Couldn't send request to peer 95304c9e18024004b2277a53318ad77e. Status: Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f846f8b3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f846f8af7cc google::LogMessage::Flush() at ??:0
@ 0x7f846f8afa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f8485a526de kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f8485a513c1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f8485a584dc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250109 20:55:50.590185 4632 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:50.590636 4632 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:50.590871 4629 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84981b978e3495fb36dc33f09cc825a. Latest consensus state: current_term: 10 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:50.591162 4629 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:50.592839 4629 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:50.593093 4629 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f8485a58359 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
I20250109 20:55:50.595535 4622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "d84981b978e3495fb36dc33f09cc825a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:50.595793 4622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:50.596954 4437 catalog_manager.cc:5813] T 00000000000000000000000000000000 P d84981b978e3495fb36dc33f09cc825a: Sending DeleteTablet for 3 replicas of tablet 0f95a0f62d3f4666adee4d1a51682f20
I20250109 20:55:50.598645 4437 catalog_manager.cc:6596] ReplaceTablet: tablet 0f95a0f62d3f4666adee4d1a51682f20 deleted and replaced by tablet 710346ad923f469799c0e126e8b6d74a
I20250109 20:55:50.598906 4437 master_service.cc:946] ReplaceTablet: received request to replace tablet b685d1fbcd744586b8e0a3024e27a4ff from {username='slave'} at 127.0.0.1:47370
I20250109 20:55:50.599592 4437 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47370:
name: "default.Table_5dcc9f9689084615b4c47a34fdd1a329"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f847260d0be kudu::Thread::SuperviseThread() at ??:0
@ 0x7f84759f06db start_thread at ??:0
@ 0x7f846e76071f clone at ??:0
I20250109 20:55:50.628834 3019 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f95a0f62d3f4666adee4d1a51682f20 with delete_type TABLET_DATA_DELETED (replaced by tablet 710346ad923f469799c0e126e8b6d74a) from {username='slave'} at 127.0.0.1:44536
I20250109 20:55:50.629494 4644 tablet_replica.cc:331] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: stopping tablet replica
I20250109 20:55:50.629703 4644 raft_consensus.cc:2238] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:55:50.630081 4644 raft_consensus.cc:2267] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:55:50.630692 4644 ts_tablet_manager.cc:1905] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:55:50.636365 4644 ts_tablet_manager.cc:1918] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:55:50.636458 4644 log.cc:1198] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: Deleting WAL directory at /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/ts-0/wal/wals/0f95a0f62d3f4666adee4d1a51682f20
I20250109 20:55:50.636765 4644 ts_tablet_manager.cc:1939] T 0f95a0f62d3f4666adee4d1a51682f20 P 0d843776b4dc4cc194def9bd50945c29: Deleting consensus metadata
W20250109 20:55:50.809638 2970 connection.cc:537] server connection from 127.0.0.1:44536 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:50.809989 4641 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:55:50.598994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.2.96.67:41259)
0109 20:55:50.627255 (+ 28261us) server_negotiation.cc:206] Beginning negotiation
0109 20:55:50.627261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0109 20:55:50.809687 (+182426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54738: BlockingRecv error: recv got EOF from 127.0.0.1:54738 (error 108)
Metrics: {"server-negotiator.queue_time_us":28139,"thread_start_us":28281,"threads_started":1}
W20250109 20:55:50.810034 4642 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:55:50.600055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.2.96.66:35185)
0109 20:55:50.626440 (+ 26385us) server_negotiation.cc:206] Beginning negotiation
0109 20:55:50.626447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0109 20:55:50.626466 (+ 19us) server_negotiation.cc:407] Connection header received
0109 20:55:50.626530 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0109 20:55:50.626534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0109 20:55:50.626609 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0109 20:55:50.626711 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0109 20:55:50.809760 (+183049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38936: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":26253,"thread_start_us":26130,"threads_started":1}
W20250109 20:55:50.810427 3368 connection.cc:537] client connection to 127.2.96.125:37271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:55:50.811070 3368 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.96.125:37271)
W20250109 20:55:50.811506 3368 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.96.125:37271)
W20250109 20:55:50.811731 3368 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.96.125:37271)
W20250109 20:55:50.811911 3368 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.125:37271)
I20250109 20:55:50.889209 4656 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:55:50.889406 4656 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:50.890107 4656 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:55:50.890931 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:55:50.891108 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:55:50.891206 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:55:50.891431 4656 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:55:51.092650 4629 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:51.092845 4629 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:51.093533 4629 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:51.095368 4529 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:51.095542 4532 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
I20250109 20:55:51.095675 4532 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e; no voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:51.095902 4629 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250109 20:55:51.672564 4661 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:51.672744 4661 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:51.673373 4661 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:51.675544 4532 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741): Network error: Client connection negotiation failed: client connection to 127.2.96.124:46741: connect: Connection refused (error 111)
W20250109 20:55:51.675760 4529 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:51.675918 4529 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9908c8a5d72498890b9058df83a1b5e; no voters: 95304c9e18024004b2277a53318ad77e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:51.676120 4661 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250109 20:55:51.854740 2433 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskljSzmY/build/asan/bin/kudu
/tmp/dist-test-taskljSzmY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.124:46741
--webserver_interface=127.2.96.124
--webserver_port=36029
--builtin_ntp_servers=127.2.96.84:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250109 20:55:52.192281 4665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:55:52.192588 4665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:55:52.192679 4665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:55:52.200414 4665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:55:52.200538 4665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:55:52.200603 4665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:55:52.200661 4665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:55:52.209978 4665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.96.84:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37605
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.96.124:46741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.96.124
--webserver_port=36029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:55:52.211737 4665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:55:52.213590 4665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:55:52.224510 4671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:52.225023 4674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:55:52.225432 4672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:55:52.225940 4665 server_base.cc:1034] running on GCE node
I20250109 20:55:52.226858 4665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:55:52.228292 4665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:55:52.229506 4665 hybrid_clock.cc:648] HybridClock initialized: now 1736456152229439 us; error 42 us; skew 500 ppm
I20250109 20:55:52.229827 4665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:55:52.238590 4665 webserver.cc:458] Webserver started at http://127.2.96.124:36029/ using document root <none> and password file <none>
I20250109 20:55:52.239333 4665 fs_manager.cc:362] Metadata directory not provided
I20250109 20:55:52.239454 4665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:55:52.244791 4665 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20250109 20:55:52.247520 4680 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:55:52.248768 4665 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250109 20:55:52.248926 4665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data,/tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
uuid: "95304c9e18024004b2277a53318ad77e"
format_stamp: "Formatted at 2025-01-09 20:55:20 on dist-test-slave-thwl"
I20250109 20:55:52.249742 4665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:55:52.272542 4665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:55:52.273460 4665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:55:52.273739 4665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:55:52.292642 4665 rpc_server.cc:307] RPC server started. Bound to: 127.2.96.124:46741
I20250109 20:55:52.292661 4731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.96.124:46741 every 8 connection(s)
I20250109 20:55:52.294178 4665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskljSzmY/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736456104686676-2433-0/minicluster-data/master-2/data/info.pb
I20250109 20:55:52.300208 4732 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:55:52.303256 4732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Bootstrap starting.
I20250109 20:55:52.303956 2433 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskljSzmY/build/asan/bin/kudu as pid 4665
I20250109 20:55:52.315884 4732 log.cc:826] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Log is configured to *not* fsync() on all Append() calls
I20250109 20:55:52.342761 3085 heartbeater.cc:346] Connected to a master server at 127.2.96.124:46741
I20250109 20:55:52.355682 4732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=48 ignored=0} inserts{seen=31 ignored=0} mutations{seen=30 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250109 20:55:52.356303 4732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: Bootstrap complete.
I20250109 20:55:52.362718 4732 raft_consensus.cc:357] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 9 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:52.365679 4732 raft_consensus.cc:738] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95304c9e18024004b2277a53318ad77e, State: Initialized, Role: FOLLOWER
I20250109 20:55:52.366479 4732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 48, Last appended: 9.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:52.367331 4738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:52.367714 4738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:52.368737 4732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:55:52.373170 3226 heartbeater.cc:346] Connected to a master server at 127.2.96.124:46741
I20250109 20:55:52.389462 4752 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:52.389655 4752 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: loading cluster ID for follower catalog manager: success
I20250109 20:55:52.393848 4752 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: acquiring CA information for follower catalog manager: success
I20250109 20:55:52.396669 4752 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:55:52.397228 4665 master_runner.cc:186] Error getting master registration for 127.2.96.125:37271: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111),
I20250109 20:55:52.406675 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:52.674209 4757 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:52.674446 4757 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:52.675166 4757 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
W20250109 20:55:52.677767 4529 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:52.684221 4707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 11 candidate_status { last_received { term: 10 index: 51 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e" is_pre_election: true
I20250109 20:55:52.684783 4707 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 9.
I20250109 20:55:52.685309 4532 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e; no voters: d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:52.685590 4757 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250109 20:55:52.685722 4757 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Starting leader election (detected failure of leader d84981b978e3495fb36dc33f09cc825a)
I20250109 20:55:52.685827 4757 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 10 FOLLOWER]: Advancing to term 11
I20250109 20:55:52.687748 4757 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:52.688319 4757 leader_election.cc:290] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 election: Requested vote from peers d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271), 95304c9e18024004b2277a53318ad77e (127.2.96.124:46741)
I20250109 20:55:52.688827 4707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9908c8a5d72498890b9058df83a1b5e" candidate_term: 11 candidate_status { last_received { term: 10 index: 51 } } ignore_live_leader: false dest_uuid: "95304c9e18024004b2277a53318ad77e"
I20250109 20:55:52.689057 4707 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 9 FOLLOWER]: Advancing to term 11
W20250109 20:55:52.689497 4529 leader_election.cc:336] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:52.694729 4707 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9908c8a5d72498890b9058df83a1b5e in term 11.
I20250109 20:55:52.695165 4532 leader_election.cc:304] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95304c9e18024004b2277a53318ad77e, b9908c8a5d72498890b9058df83a1b5e; no voters: d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:52.695475 4757 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 11 FOLLOWER]: Leader election won for term 11
I20250109 20:55:52.695940 4757 raft_consensus.cc:695] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [term 11 LEADER]: Becoming Leader. State: Replica: b9908c8a5d72498890b9058df83a1b5e, State: Running, Role: LEADER
I20250109 20:55:52.696367 4757 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 10.51, Last appended by leader: 51, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:52.697827 4761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 11 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:52.698078 4761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: LEADER
I20250109 20:55:52.698745 4763 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:55:52.702577 4763 catalog_manager.cc:670] Loaded metadata for table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c]
I20250109 20:55:52.703621 4763 catalog_manager.cc:670] Loaded metadata for table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06]
I20250109 20:55:52.704221 4763 catalog_manager.cc:670] Loaded metadata for table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec]
I20250109 20:55:52.704792 4763 catalog_manager.cc:670] Loaded metadata for table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc]
I20250109 20:55:52.705425 4763 catalog_manager.cc:670] Loaded metadata for table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869]
I20250109 20:55:52.706043 4763 catalog_manager.cc:670] Loaded metadata for table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31]
I20250109 20:55:52.706655 4763 catalog_manager.cc:670] Loaded metadata for table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702]
I20250109 20:55:52.707267 4763 catalog_manager.cc:670] Loaded metadata for table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc]
I20250109 20:55:52.707793 4763 catalog_manager.cc:670] Loaded metadata for table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3]
I20250109 20:55:52.710855 4763 tablet_loader.cc:96] loaded metadata for tablet 07334099f3184a179000213a1839dddf (table default.table_320b149b7d80477bb465bb9b3351d0be [id=f19741d30cad406b81ee6067dc94b702])
I20250109 20:55:52.712515 4763 tablet_loader.cc:96] loaded metadata for tablet 1010e1aeb5d140bf9f29d386bc195ed7 (table default.table_edccb187ef7149b1a1d158edc0924690 [id=f9c3ac24970e4e47af6934218b3e18cc])
I20250109 20:55:52.712956 4763 tablet_loader.cc:96] loaded metadata for tablet 2b5cf5740eb642f6aae83df3916cbf3f (table default.table_94dbbe2d395441859b039d900093b7a2 [id=83f5ad9633e74385904e3e6cfa452ecc])
I20250109 20:55:52.713404 4763 tablet_loader.cc:96] loaded metadata for tablet 45574543f3dd492c852637bd9739fcf3 (table default.table_e27d0bbbd6a445f099a3be8e238bf9c8 [id=095b9b29244f4bce82fe2b798a9e2a0c])
I20250109 20:55:52.714149 4763 tablet_loader.cc:96] loaded metadata for tablet 710346ad923f469799c0e126e8b6d74a (table default.table_f5f640c97c72444396c24fbb34b9ec18 [id=c96a98e7e45749cb8ed5ffdfab329c31])
I20250109 20:55:52.714672 4764 raft_consensus.cc:491] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:55:52.714846 4763 tablet_loader.cc:96] loaded metadata for tablet 7df6e6361c074dc3984dbe5d92b1205d (table default.table_da5d7a22c7704d4facb0ae576942a0d2 [id=6ef574c52a36423db5b9b06b9909f0ec])
I20250109 20:55:52.715024 4764 raft_consensus.cc:513] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:52.715229 4763 tablet_loader.cc:96] loaded metadata for tablet 83ac24dec39543e7a617bf4fc3366c10 (table default.table_d2f3663d9ed949609d925f9bb37f7633 [id=fd4270c63e4c4ba1a145c0e996e16fb3])
I20250109 20:55:52.716722 4763 tablet_loader.cc:96] loaded metadata for tablet c17492c45ef04ace967a0cd0df3a8eaf (table default.table_84db4e769e42410f985d1d5f5cbe527a [id=c6228f0aa09b45e2b416a88a82683869])
I20250109 20:55:52.716666 3180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f95a0f62d3f4666adee4d1a51682f20" candidate_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd7534821da847d0bf53f35562f4b653" is_pre_election: true
I20250109 20:55:52.717074 4763 tablet_loader.cc:96] loaded metadata for tablet c5339f2d6a1547c3aad25d50455eeab2 (table default.table_cbd0b67985f24940af88114907c635eb [id=206fdcbf4e334b578acb0dc524d09f06])
W20250109 20:55:52.717576 3251 leader_election.cc:343] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20
I20250109 20:55:52.717792 3251 leader_election.cc:304] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:55:52.718317 4763 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:55:52.718735 4764 leader_election.cc:290] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
I20250109 20:55:52.719115 4764 raft_consensus.cc:2743] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:55:52.720306 4763 catalog_manager.cc:1260] Loaded cluster ID: 9b1eeba70c4a4d008ed5999b7058f7fb
I20250109 20:55:52.720450 4763 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:55:52.723124 4763 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:55:52.724021 4763 catalog_manager.cc:5910] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e: Loaded TSK: 0
I20250109 20:55:52.724917 4763 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:55:52.725821 4763 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250109 20:55:52.726754 4763 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250109 20:55:52.763235 3361 heartbeater.cc:346] Connected to a master server at 127.2.96.124:46741
I20250109 20:55:52.765080 4696 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" instance_seqno: 1736456122600579) as {username='slave'} at 127.2.96.67:38927; Asking this server to re-register.
I20250109 20:55:52.765619 3361 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:52.767066 4696 ts_manager.cc:194] Registered new tserver with Master: 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:52.787083 4541 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38818:
table { table_name: "default.Table_94dbbe2d395441859b039d900093b7a2" } modify_external_catalogs: true
I20250109 20:55:52.787482 4541 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38818:
table { table_name: "default.Table_94dbbe2d395441859b039d900093b7a2" } modify_external_catalogs: true
F20250109 20:55:52.788257 4599 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_cbd0b67985f24940af88114907c635eb
*** Check failure stack trace: ***
I20250109 20:55:52.788471 4539 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38818:
name: "default.Table_e992250ce97b4de3ac9cf07ff61775a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:52.788909 4544 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f95a0f62d3f4666adee4d1a51682f20 from {username='slave'} at 127.0.0.1:38818
I20250109 20:55:52.790030 4545 master_service.cc:946] ReplaceTablet: received request to replace tablet d8d7677ecbb9460192ca4323d38b03ad from {username='slave'} at 127.0.0.1:38818
I20250109 20:55:52.791231 4544 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38818:
name: "default.Table_0d2dc89f52bc4afc8d3de54485a3de62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:52.791260 4545 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38818:
table { table_name: "default.Table_e27d0bbbd6a445f099a3be8e238bf9c8" } new_table_name: "default.Table_c66ec467f4264b9bbd248d748ba125c0" modify_external_catalogs: true
I20250109 20:55:52.791184 4542 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38818:
table { table_name: "default.Table_cbd0b67985f24940af88114907c635eb" } new_table_name: "default.Table_61a48f808b814c4aadbc420991d9db06" modify_external_catalogs: true
I20250109 20:55:52.792454 4543 master_service.cc:946] ReplaceTablet: received request to replace tablet 8a47d3e2c8ea4690bbdbe87936af9415 from {username='slave'} at 127.0.0.1:38818
I20250109 20:55:52.792119 4538 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38818:
table { table_name: "default.Table_edccb187ef7149b1a1d158edc0924690" } new_table_name: "default.Table_1b53d8836f984435902fbc1c6a9073b1" modify_external_catalogs: true
I20250109 20:55:52.791623 4536 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38818:
name: "default.Table_e672267cd18e4969b28037e40d7bc5cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:52.791499 4540 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38818:
name: "default.Table_20e2171c365c4c3e8aafa501e51dae5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:55:52.793191 4543 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38818:
table { table_name: "default.Table_320b149b7d80477bb465bb9b3351d0be" } new_table_name: "default.Table_9c2e38f90afc4b9dbf223d20ffa931fb" modify_external_catalogs: true
I20250109 20:55:52.791093 4537 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38818:
name: "default.Table_66e7788f7db54a2ea368c19b84195df5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f52d05c4dcd google::LogMessage::Fail() at ??:0
@ 0x7f52d05c8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f52d05c47cc google::LogMessage::Flush() at ??:0
@ 0x7f52d05c4a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f52e67676de kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250109 20:55:52.807946 4707 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Refusing update from remote peer b9908c8a5d72498890b9058df83a1b5e: Log matching property violated. Preceding OpId in replica: term: 9 index: 49. Preceding OpId from leader: term: 11 index: 53. (index mismatch)
I20250109 20:55:52.808737 4761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 48, Time since last communication: 0.000s
W20250109 20:55:52.809427 4529 consensus_peers.cc:487] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e -> Peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Couldn't send request to peer d84981b978e3495fb36dc33f09cc825a. Status: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f52e67663c1 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f52e676d4dc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f52e676d359 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
I20250109 20:55:52.826150 4738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9908c8a5d72498890b9058df83a1b5e. Latest consensus state: current_term: 11 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:52.826361 4738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:52.827085 4737 mvcc.cc:204] Tried to move back new op lower bound from 7112524392721031168 to 7112524392497971200. Current Snapshot: MvccSnapshot[applied={T|T < 7112524392721031168 or (T in {7112524392721031168})}]
I20250109 20:55:52.827724 4768 mvcc.cc:204] Tried to move back new op lower bound from 7112524401893425152 to 7112524401447059456. Current Snapshot: MvccSnapshot[applied={T|T < 7112524401893425152}]
@ 0x7f52d33220be kudu::Thread::SuperviseThread() at ??:0
I20250109 20:55:52.829337 4738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:52.829569 4738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:55:52.826822 4773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:52.831429 4773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f52d67056db start_thread at ??:0
I20250109 20:55:52.832248 4761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "b9908c8a5d72498890b9058df83a1b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } } }
I20250109 20:55:52.832532 4761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9908c8a5d72498890b9058df83a1b5e [sys.catalog]: This master's current role is: LEADER
@ 0x7f52cf47571f clone at ??:0
I20250109 20:55:52.846944 4696 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" instance_seqno: 1736456121613668) as {username='slave'} at 127.2.96.65:33049; Asking this server to re-register.
I20250109 20:55:52.847864 3085 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:52.849530 4696 ts_manager.cc:194] Registered new tserver with Master: 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615)
I20250109 20:55:52.876387 4696 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cd7534821da847d0bf53f35562f4b653" instance_seqno: 1736456122119526) as {username='slave'} at 127.2.96.66:33003; Asking this server to re-register.
I20250109 20:55:52.876986 3226 heartbeater.cc:463] Registering TS with master...
I20250109 20:55:52.878265 4696 ts_manager.cc:194] Registered new tserver with Master: cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:55:52.989688 3371 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:52.989974 3371 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:52.990233 3371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.126:37541)
W20250109 20:55:52.990523 3371 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.96.126:37541)
I20250109 20:55:53.251013 4785 raft_consensus.cc:491] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:55:53.251188 4785 raft_consensus.cc:513] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:53.251950 4785 leader_election.cc:290] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:53.252446 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f95a0f62d3f4666adee4d1a51682f20" candidate_uuid: "cd7534821da847d0bf53f35562f4b653" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" is_pre_election: true
I20250109 20:55:53.252713 3316 raft_consensus.cc:2463] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7534821da847d0bf53f35562f4b653 in term 1.
W20250109 20:55:53.252748 3116 leader_election.cc:343] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20
I20250109 20:55:53.253068 3117 leader_election.cc:304] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, cd7534821da847d0bf53f35562f4b653; no voters: 0d843776b4dc4cc194def9bd50945c29
I20250109 20:55:53.253284 4785 raft_consensus.cc:2798] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250109 20:55:53.253374 4785 raft_consensus.cc:491] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:55:53.253436 4785 raft_consensus.cc:3054] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:55:53.255327 4785 raft_consensus.cc:513] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:53.255818 4785 leader_election.cc:290] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 2 election: Requested vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), 0aab39ff46cf459e8ba528e5356ab3b6 (127.2.96.67:41259)
I20250109 20:55:53.256246 3316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f95a0f62d3f4666adee4d1a51682f20" candidate_uuid: "cd7534821da847d0bf53f35562f4b653" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0aab39ff46cf459e8ba528e5356ab3b6"
I20250109 20:55:53.256424 3316 raft_consensus.cc:3054] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Advancing to term 2
I20250109 20:55:53.258194 3316 raft_consensus.cc:2463] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7534821da847d0bf53f35562f4b653 in term 2.
I20250109 20:55:53.258556 3117 leader_election.cc:304] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6, cd7534821da847d0bf53f35562f4b653; no voters:
W20250109 20:55:53.258733 3116 leader_election.cc:343] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20
I20250109 20:55:53.258824 4785 raft_consensus.cc:2798] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 2 FOLLOWER]: Leader election won for term 2
I20250109 20:55:53.259155 4785 raft_consensus.cc:695] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [term 2 LEADER]: Becoming Leader. State: Replica: cd7534821da847d0bf53f35562f4b653, State: Running, Role: LEADER
I20250109 20:55:53.259454 3227 heartbeater.cc:502] Master 127.2.96.126:37541 was elected leader, sending a full tablet report...
I20250109 20:55:53.259430 4785 consensus_queue.cc:237] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:53.283056 3086 heartbeater.cc:502] Master 127.2.96.126:37541 was elected leader, sending a full tablet report...
I20250109 20:55:53.283358 3362 heartbeater.cc:502] Master 127.2.96.126:37541 was elected leader, sending a full tablet report...
I20250109 20:55:53.329130 4773 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:53.329332 4773 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:53.331269 4773 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:53.331965 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:53.333177 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:53.333348 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:53.333534 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:53.333822 4773 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:55:53.923568 4791 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:53.923766 4791 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:53.924487 4791 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:53.926319 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:53.926476 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:53.926615 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:53.926867 4791 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:55:54.073213 3316 raft_consensus.cc:1270] T 0f95a0f62d3f4666adee4d1a51682f20 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 2 FOLLOWER]: Refusing update from remote peer cd7534821da847d0bf53f35562f4b653: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250109 20:55:54.073812 4795 consensus_queue.cc:1035] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250109 20:55:54.079874 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:55:54.462863 4803 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:54.463063 4803 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:54.463798 4803 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:54.465682 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:54.465837 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:54.465945 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:54.466221 4803 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:55:54.544195 2433 master-stress-test.cc:577] Killing test thread 1/25
I20250109 20:55:55.116194 4816 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:55.116374 4816 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:55.116950 4816 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:55.118531 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:55.118675 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:55.118786 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:55.119005 4816 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:55:56.236653 4827 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:56.236883 4827 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:56.237463 4827 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:56.239662 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:55:56.239826 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:55:56.239964 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:56.240200 4827 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:55:56.624506 4835 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:55:56.624665 4835 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:55:56.625385 4835 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:55:56.626173 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:55:56.626351 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:55:56.626495 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:55:56.626705 4835 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:55:57.408048 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:55:57.408207 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:55:58.813177 4856 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:55:58.813381 4856 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:55:58.813985 4856 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:55:58.814960 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250109 20:55:58.816048 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:55:58.816212 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:55:58.816361 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:55:58.816594 4856 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:56:01.888216 4891 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:56:01.888429 4891 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:56:01.889305 4891 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:56:01.890059 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:56:01.890209 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:56:01.890319 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:56:01.890564 4891 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:56:02.409616 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:02.409817 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:56:03.957250 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 6: this message will repeat every 5th retry.
I20250109 20:56:04.291666 4913 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:56:04.291891 4913 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:56:04.292589 4913 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:56:04.293828 4681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:56:04.295207 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:04.295395 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:56:04.295527 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:56:04.295785 4913 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:56:07.411190 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:07.411325 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:56:08.602607 4959 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:56:08.602816 4959 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:56:08.603488 4959 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:56:08.605823 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:08.605988 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:56:08.606105 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:56:08.606341 4959 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:56:12.377558 5001 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:56:12.377775 5001 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:56:12.378551 5001 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:56:12.379478 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250109 20:56:12.380584 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:56:12.380738 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:56:12.380867 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:56:12.381114 5001 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:56:12.412631 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:12.412833 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:56:14.009058 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 11: this message will repeat every 5th retry.
I20250109 20:56:15.167979 5032 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:56:15.168200 5032 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:56:15.168941 5032 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:56:15.169869 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:56:15.170029 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:56:15.170186 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:56:15.170469 5032 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:56:17.414076 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:17.414235 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:56:22.415724 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:22.415902 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:56:22.501468 5104 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:56:22.501685 5104 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:56:22.502629 5104 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:56:22.503482 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:56:22.503650 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:56:22.503784 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:56:22.504019 5104 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:56:23.182562 3362 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (61 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:23.350590 3086 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (60 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:23.432265 3227 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (76 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:24.650529 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 16: this message will repeat every 5th retry.
I20250109 20:56:27.417424 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:27.417588 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:56:28.471946 5167 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:56:28.472138 5167 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:56:28.473039 5167 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:56:28.474004 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:56:28.475111 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:28.475245 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:56:28.475343 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:56:28.475548 5167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:56:32.419229 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:32.419392 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:56:35.149276 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 21: this message will repeat every 5th retry.
I20250109 20:56:36.098601 5248 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:56:36.098803 5248 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:56:36.099504 5248 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:56:36.100353 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:56:36.100533 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:56:36.100652 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:56:36.100927 5248 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:56:36.687386 5257 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:56:36.687594 5257 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:56:36.688295 5257 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:56:36.689188 4681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:56:36.690198 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:36.690341 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:56:36.690474 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:56:36.690701 5257 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:56:37.423988 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:37.424144 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:56:39.050083 5287 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:56:39.050289 5287 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:56:39.050948 5287 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:56:39.052893 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:39.053077 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:56:39.053193 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:56:39.053390 5287 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:56:42.425629 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:42.425801 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:56:44.958781 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 26: this message will repeat every 5th retry.
I20250109 20:56:47.427289 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:47.427435 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:56:52.428771 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:52.428977 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:56:55.138743 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 31: this message will repeat every 5th retry.
I20250109 20:56:55.522423 5423 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:56:55.522605 5423 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:56:55.523384 5423 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:56:55.524307 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:56:55.524470 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:56:55.524556 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:56:55.524762 5423 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:56:55.827134 5427 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:56:55.827364 5427 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:56:55.828207 5427 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:56:55.829739 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250109 20:56:55.831301 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:55.831449 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:56:55.831542 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:56:55.831750 5427 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:56:56.542553 5437 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:56:56.542755 5437 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:56:56.543329 5437 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:56:56.545639 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:56:56.545785 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:56:56.545892 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:56:56.546115 5437 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:56:57.430330 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:56:57.430518 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:02.431969 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:02.432132 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:02.926575 5506 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:57:02.927256 5506 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:57:02.927896 5506 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:57:02.928462 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250109 20:57:02.929530 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:57:02.929677 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:57:02.929771 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:57:02.929971 5506 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250109 20:57:04.758844 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 36: this message will repeat every 5th retry.
I20250109 20:57:06.175026 5540 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:57:06.175205 5540 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:57:06.175890 5540 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:57:06.176779 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:57:06.176942 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:57:06.177070 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:57:06.177294 5540 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:57:07.433372 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:07.433518 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:09.533380 5574 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:57:09.533574 5574 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:57:09.534263 5574 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:57:09.535147 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:57:09.535312 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:57:09.535426 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:57:09.535638 5574 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:57:12.434813 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:12.434974 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:14.333523 5626 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:57:14.333714 5626 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:57:14.334372 5626 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:57:14.335325 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:57:14.336397 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:57:14.336556 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:57:14.336665 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:57:14.336885 5626 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250109 20:57:14.838351 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 41: this message will repeat every 5th retry.
I20250109 20:57:15.405470 5641 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:57:15.405719 5641 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:57:15.406512 5641 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:57:15.408313 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:57:15.408473 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:57:15.408607 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:57:15.408869 5641 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:57:17.436151 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:17.436322 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:19.556898 5689 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:57:19.557096 5689 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:57:19.557989 5689 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:57:19.558883 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:57:19.559044 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:57:19.559182 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:57:19.559415 5689 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:57:22.437527 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:22.437671 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:57:23.349701 3363 heartbeater.cc:643] Failed to heartbeat to 127.2.96.125:37271 (186 consecutive failures): Network error: Failed to ping master at 127.2.96.125:37271: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:57:23.482175 3086 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (180 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:57:23.648706 3228 heartbeater.cc:643] Failed to heartbeat to 127.2.96.125:37271 (231 consecutive failures): Network error: Failed to ping master at 127.2.96.125:37271: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:57:23.997885 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 46: this message will repeat every 5th retry.
I20250109 20:57:27.438970 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:27.439177 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:29.059863 5781 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:57:29.060075 5781 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:57:29.060833 5781 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:57:29.061563 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:57:29.061739 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:57:29.061856 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:57:29.062078 5781 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:57:32.142143 5812 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:57:32.142359 5812 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:57:32.143241 5812 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:57:32.144148 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250109 20:57:32.145195 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:57:32.145344 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:57:32.145459 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:57:32.145684 5812 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:57:32.440145 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:32.440317 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:33.748003 5832 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:57:33.748189 5832 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:57:33.748800 5832 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:57:33.749572 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:57:33.749707 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:57:33.749842 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:57:33.750036 5832 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:57:33.845206 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 51: this message will repeat every 5th retry.
I20250109 20:57:37.441910 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:37.442066 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:42.443612 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:42.443814 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:44.265709 5944 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:57:44.265926 5944 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:57:44.267341 5944 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:57:44.268211 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:57:44.268362 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:57:44.268514 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:57:44.268735 5944 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:57:44.576741 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 56: this message will repeat every 5th retry.
I20250109 20:57:47.368674 5978 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:57:47.368885 5978 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:57:47.369768 5978 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:57:47.370831 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:57:47.372015 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:57:47.372264 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:57:47.372399 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:57:47.372622 5978 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:57:47.445369 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:47.445565 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:52.446981 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:52.447154 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:57:52.447465 4686 maintenance_manager.cc:418] P 95304c9e18024004b2277a53318ad77e: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033356
I20250109 20:57:52.499814 4685 maintenance_manager.cc:642] P 95304c9e18024004b2277a53318ad77e: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.052s user 0.019s sys 0.005s Metrics: {"bytes_written":15397,"cfile_init":1,"dirs.queue_time_us":386,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1075,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":548,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":33,"thread_start_us":516,"threads_started":2,"wal-append.queue_time_us":368}
I20250109 20:57:52.500923 4686 maintenance_manager.cc:418] P 95304c9e18024004b2277a53318ad77e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1736 bytes on disk
I20250109 20:57:52.501541 4685 maintenance_manager.cc:642] P 95304c9e18024004b2277a53318ad77e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20250109 20:57:54.006446 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 61: this message will repeat every 5th retry.
I20250109 20:57:54.307034 6053 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:57:54.307250 6053 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:57:54.308080 6053 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:57:54.309002 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:57:54.309209 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:57:54.309340 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:57:54.309643 6053 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:57:57.448518 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:57:57.448712 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:58:02.450479 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:02.450676 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:58:04.571223 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 66: this message will repeat every 5th retry.
I20250109 20:58:05.067746 6192 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:58:05.067955 6192 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:58:05.068821 6192 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:58:05.069885 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:58:05.071337 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:58:05.071666 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:58:05.071784 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:58:05.072057 6192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:58:07.452042 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:07.452195 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:58:08.378046 6247 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:58:08.378218 6247 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:58:08.378988 6247 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:58:08.379793 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:58:08.379937 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:58:08.380081 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:58:08.380370 6247 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:58:08.802141 6254 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:58:08.802305 6254 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:58:08.803032 6254 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:58:08.804739 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:58:08.804883 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:58:08.805008 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:58:08.805214 6254 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:58:10.376093 6279 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:58:10.376329 6279 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:58:10.376969 6279 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:58:10.377897 4681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250109 20:58:10.378815 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:58:10.378969 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:58:10.379077 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:58:10.379331 6279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:58:12.453624 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:12.453819 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:58:14.783371 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 71: this message will repeat every 5th retry.
I20250109 20:58:17.455400 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:17.455585 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:58:21.027730 6431 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:58:21.027956 6431 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:58:21.028633 6431 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:58:21.029516 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:58:21.031229 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:58:21.031391 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:58:21.031505 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:58:21.031744 6431 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:58:22.457074 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:22.457247 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:58:23.515223 3362 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (301 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:58:23.614233 3086 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (300 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:58:23.756904 3227 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (378 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:58:23.913333 6471 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:58:23.913550 6471 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:58:23.914448 6471 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:58:23.915562 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:58:23.915736 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:58:23.915844 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:58:23.916069 6471 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:58:24.416635 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 76: this message will repeat every 5th retry.
I20250109 20:58:27.458619 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:27.458810 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:58:27.888527 6526 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:58:27.888739 6526 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:58:27.889394 6526 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:58:27.889956 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:58:27.891042 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:58:27.891228 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:58:27.891342 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:58:27.891539 6526 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:58:28.654856 6539 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:58:28.655064 6539 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:58:28.655691 6539 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:58:28.656499 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:58:28.656653 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:58:28.656774 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:58:28.657016 6539 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:58:32.460346 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:32.460536 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:58:34.748653 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 81: this message will repeat every 5th retry.
I20250109 20:58:37.461712 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:37.461867 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:58:39.532317 6694 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:58:39.532543 6694 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:58:39.533376 6694 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:58:39.534204 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:58:39.534338 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:58:39.534530 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:58:39.534761 6694 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:58:42.463640 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:42.463797 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:58:42.996600 6744 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:58:42.997220 6744 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:58:42.997833 6744 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:58:42.998754 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:58:42.998904 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:58:42.999002 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:58:42.999181 6744 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:58:44.726372 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 86: this message will repeat every 5th retry.
I20250109 20:58:46.318549 6787 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:58:46.318761 6787 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:58:46.319660 6787 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:58:46.321853 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:58:46.323266 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:58:46.323421 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:58:46.323596 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:58:46.324007 6787 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:58:47.465440 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:47.465657 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:58:48.243907 6821 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:58:48.244155 6821 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:58:48.245054 6821 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:58:48.245683 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:58:48.246008 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:58:48.246114 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:58:48.246362 6821 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:58:52.438999 6880 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:58:52.439172 6880 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:58:52.439761 6880 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:58:52.440575 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:58:52.441679 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:58:52.441804 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:58:52.441902 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:58:52.442087 6880 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:58:52.466902 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:52.467051 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:58:52.969161 6888 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:58:52.969359 6888 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:58:52.970057 6888 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:58:52.970916 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:58:52.971099 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:58:52.971207 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:58:52.971412 6888 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:58:55.393078 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 91: this message will repeat every 5th retry.
I20250109 20:58:56.087668 6933 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:58:56.087869 6933 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:58:56.088670 6933 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:58:56.089357 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:58:56.089493 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:58:56.089637 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:58:56.089871 6933 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:58:57.468371 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:58:57.468537 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:59:01.667919 7006 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:59:01.668113 7006 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:59:01.668918 7006 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:59:01.669840 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:59:01.670004 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:59:01.670166 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:59:01.670429 7006 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:59:02.469897 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:02.470075 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:59:04.794752 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 96: this message will repeat every 5th retry.
I20250109 20:59:07.471884 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:07.472070 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:59:11.553557 7137 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:59:11.553800 7137 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:59:11.554590 7137 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:59:11.555465 4681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 20:59:11.556452 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:59:11.556599 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:59:11.556697 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:59:11.556900 7137 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:59:12.473803 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:12.473978 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:59:13.491473 7165 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:59:13.491678 7165 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:59:13.492218 7165 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:59:13.496073 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:59:13.496462 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:59:13.496605 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:59:13.496850 7165 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:59:13.518957 7172 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:59:13.519135 7172 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:59:13.519912 7172 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:59:13.520731 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:59:13.520876 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:59:13.520973 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:59:13.521174 7172 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:59:14.325058 7182 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:59:14.325266 7182 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:59:14.326030 7182 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:59:14.329116 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:59:14.329603 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:59:14.329705 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:59:14.329923 7182 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250109 20:59:14.418329 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 101: this message will repeat every 5th retry.
I20250109 20:59:17.475373 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:17.475553 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:59:21.667299 7255 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:59:21.667500 7255 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:59:21.668306 7255 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:59:21.669163 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:59:21.669363 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:59:21.669518 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:59:21.669757 7255 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:59:22.476758 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:22.476922 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:59:23.678647 3363 heartbeater.cc:643] Failed to heartbeat to 127.2.96.125:37271 (426 consecutive failures): Network error: Failed to ping master at 127.2.96.125:37271: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:59:23.744210 3086 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (420 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:59:23.808719 3228 heartbeater.cc:643] Failed to heartbeat to 127.2.96.125:37271 (531 consecutive failures): Network error: Failed to ping master at 127.2.96.125:37271: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:59:24.068833 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 106: this message will repeat every 5th retry.
I20250109 20:59:27.478303 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:27.478506 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:59:31.184381 7385 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:59:31.184562 7385 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:59:31.185364 7385 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:59:31.186266 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250109 20:59:31.187743 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:59:31.187914 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:59:31.188045 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:59:31.188311 7385 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:59:32.262513 7403 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:59:32.262732 7403 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:59:32.263401 7403 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:59:32.265134 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 20:59:32.265297 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
I20250109 20:59:32.265441 4681 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:59:32.265650 7403 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:59:32.479782 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:32.479950 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:59:34.810097 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 111: this message will repeat every 5th retry.
I20250109 20:59:36.069437 7453 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:59:36.069660 7453 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:59:36.070497 7453 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:59:36.071327 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:59:36.071494 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:59:36.071589 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:59:36.071787 7453 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:59:37.481518 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:37.481665 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:59:42.183676 7532 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:59:42.183857 7532 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:59:42.184531 7532 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:59:42.185235 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:59:42.185386 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:59:42.185525 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:59:42.185748 7532 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:59:42.483139 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:42.483348 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 20:59:44.507810 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 116: this message will repeat every 5th retry.
I20250109 20:59:47.484788 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:47.484956 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:59:49.130947 7625 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 20:59:49.131131 7625 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 20:59:49.131855 7625 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 20:59:49.132705 4681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250109 20:59:49.133944 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 20:59:49.134068 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 20:59:49.134191 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 20:59:49.134444 7625 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 20:59:52.486477 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:52.486644 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:59:53.646649 7686 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 20:59:53.646835 7686 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 20:59:53.647732 7686 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 20:59:53.648525 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 20:59:53.648679 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 20:59:53.648809 3250 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 20:59:53.649020 7686 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:59:53.924506 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 121: this message will repeat every 5th retry.
I20250109 20:59:57.488065 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 20:59:57.488234 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 21:00:01.165377 7780 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 21:00:01.165593 7780 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 21:00:01.166268 7780 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 21:00:01.167351 4681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 21:00:01.168664 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 21:00:01.168823 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 21:00:01.168921 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 21:00:01.169131 7780 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250109 21:00:02.489686 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 21:00:02.489851 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250109 21:00:03.302417 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 126: this message will repeat every 5th retry.
I20250109 21:00:07.491313 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 21:00:07.491462 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 21:00:12.493175 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 21:00:12.493333 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 21:00:12.846165 7934 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 21:00:12.846354 7934 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 21:00:12.847203 7934 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 21:00:12.847997 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 21:00:12.848174 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 21:00:12.848315 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 21:00:12.848526 7934 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 21:00:13.007015 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 131: this message will repeat every 5th retry.
I20250109 21:00:17.494851 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 21:00:17.495048 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 21:00:18.029814 8003 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9908c8a5d72498890b9058df83a1b5e)
I20250109 21:00:18.030021 8003 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9908c8a5d72498890b9058df83a1b5e" member_type: VOTER last_known_addr { host: "127.2.96.126" port: 37541 } } peers { permanent_uuid: "d84981b978e3495fb36dc33f09cc825a" member_type: VOTER last_known_addr { host: "127.2.96.125" port: 37271 } } peers { permanent_uuid: "95304c9e18024004b2277a53318ad77e" member_type: VOTER last_known_addr { host: "127.2.96.124" port: 46741 } }
I20250109 21:00:18.030839 8003 leader_election.cc:290] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541), d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271)
W20250109 21:00:18.031666 4684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250109 21:00:18.032598 4681 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer d84981b978e3495fb36dc33f09cc825a (127.2.96.125:37271): Network error: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 21:00:18.032788 4684 leader_election.cc:336] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b9908c8a5d72498890b9058df83a1b5e (127.2.96.126:37541): Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
I20250109 21:00:18.032903 4684 leader_election.cc:304] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95304c9e18024004b2277a53318ad77e; no voters: b9908c8a5d72498890b9058df83a1b5e, d84981b978e3495fb36dc33f09cc825a
I20250109 21:00:18.033152 8003 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 95304c9e18024004b2277a53318ad77e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250109 21:00:22.378159 3116 consensus_peers.cc:487] T 0f95a0f62d3f4666adee4d1a51682f20 P cd7534821da847d0bf53f35562f4b653 -> Peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Couldn't send request to peer 0d843776b4dc4cc194def9bd50945c29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f95a0f62d3f4666adee4d1a51682f20. This is attempt 136: this message will repeat every 5th retry.
I20250109 21:00:22.496416 4665 master_runner.cc:186] Error getting master registration for 127.2.96.126:37541: Network error: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111),
I20250109 21:00:22.496584 4665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 21:00:23.451454 8075 raft_consensus.cc:491] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d843776b4dc4cc194def9bd50945c29)
I20250109 21:00:23.451638 8075 raft_consensus.cc:513] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d843776b4dc4cc194def9bd50945c29" member_type: VOTER last_known_addr { host: "127.2.96.65" port: 33615 } } peers { permanent_uuid: "cd7534821da847d0bf53f35562f4b653" member_type: VOTER last_known_addr { host: "127.2.96.66" port: 35185 } } peers { permanent_uuid: "0aab39ff46cf459e8ba528e5356ab3b6" member_type: VOTER last_known_addr { host: "127.2.96.67" port: 41259 } }
I20250109 21:00:23.452335 8075 leader_election.cc:290] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615), cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185)
W20250109 21:00:23.453261 3250 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd7534821da847d0bf53f35562f4b653 (127.2.96.66:35185): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
W20250109 21:00:23.453446 3251 leader_election.cc:343] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d843776b4dc4cc194def9bd50945c29 (127.2.96.65:33615): Not found: Tablet not found: 84eb8b3ef2b0414da0e9de1ba8c4a129
I20250109 21:00:23.453574 3251 leader_election.cc:304] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0aab39ff46cf459e8ba528e5356ab3b6; no voters: 0d843776b4dc4cc194def9bd50945c29, cd7534821da847d0bf53f35562f4b653
I20250109 21:00:23.453799 8075 raft_consensus.cc:2743] T 84eb8b3ef2b0414da0e9de1ba8c4a129 P 0aab39ff46cf459e8ba528e5356ab3b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 21:00:23.843658 3363 heartbeater.cc:643] Failed to heartbeat to 127.2.96.125:37271 (546 consecutive failures): Network error: Failed to ping master at 127.2.96.125:37271: Client connection negotiation failed: client connection to 127.2.96.125:37271: connect: Connection refused (error 111)
W20250109 21:00:23.870355 3086 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (540 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 21:00:23.910113 3227 heartbeater.cc:643] Failed to heartbeat to 127.2.96.126:37541 (673 consecutive failures): Network error: Failed to ping master at 127.2.96.126:37541: Client connection negotiation failed: client connection to 127.2.96.126:37541: connect: Connection refused (error 111)
W20250109 21:00:23.910794 3371 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
W20250109 21:00:23.911020 3371 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
W20250109 21:00:23.911120 3371 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
W20250109 21:00:23.911187 3371 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
F20250109 21:00:23.921970 3385 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5dcc9f9689084615b4c47a34fdd1a329 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
*** Aborted at 1736456423 (unix time) try "date -d @1736456423" if you are using GNU date ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
W20250109 21:00:23.926086 3371 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.2.96.126:37541)
W20250109 21:00:23.926354 3371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.96.126:37541)
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927542 3386 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927542 3386 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927605 3393 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927542 3386 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927605 3393 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928086 3387 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927542 3386 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927605 3393 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928086 3387 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928503 3394 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927542 3386 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927605 3393 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928086 3387 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928503 3394 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928598 3381 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2132f850190647e6af0ace1b48daa3ac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927542 3386 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927605 3393 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928086 3387 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928503 3394 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928598 3381 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2132f850190647e6af0ace1b48daa3ac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928946 3390 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000981) received by PID 2433 (TID 0x7f5c12eab700) from PID 2433; stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927542 3386 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927605 3393 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928086 3387 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928503 3394 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928598 3381 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2132f850190647e6af0ace1b48daa3ac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928946 3390 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.929589 3382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48885eaf759344b8a2b07e6b94fa13ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7f5c29fb6980 (unknown) at ??:0
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927542 3386 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927605 3393 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928086 3387 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928503 3394 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928598 3381 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2132f850190647e6af0ace1b48daa3ac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928946 3390 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.929589 3382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48885eaf759344b8a2b07e6b94fa13ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.931449 3388 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
F20250109 21:00:23.922003 3384 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e992250ce97b4de3ac9cf07ff61775a8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.923661 3392 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.924340 3383 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e672267cd18e4969b28037e40d7bc5cf on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925067 3391 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.925172 3389 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927224 3380 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_20e2171c365c4c3e8aafa501e51dae5a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927542 3386 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.927605 3393 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928086 3387 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928503 3394 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928598 3381 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2132f850190647e6af0ace1b48daa3ac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.928946 3390 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.929589 3382 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_48885eaf759344b8a2b07e6b94fa13ed on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.931449 3388 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)F20250109 21:00:23.931463 3378 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_66e7788f7db54a2ea368c19b84195df5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 732) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.2.96.126:37541,127.2.96.125:37271,127.2.96.124:46741, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7f5c22fc6fb7 gsignal at ??:0
@ 0x7f5c22fc8921 abort at ??:0
@ 0x7f5c25cefdcd google::LogMessage::Fail() at ??:0
@ 0x7f5c25cf3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5c25cef7cc google::LogMessage::Flush() at ??:0
@ 0x7f5c25cf0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x4085b2 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:297
@ 0x447a84 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44793d std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x4478fa _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x4478ae std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x4475e9 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f5c2364e6df (unknown) at ??:0
@ 0x7f5c29fab6db start_thread at ??:0
@ 0x7f5c230a971f clone at ??:0