Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 08:04:09.168946 1101 test_util.cc:276] Using random seed: -474078176
2025-06-25T08:04:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T08:04:09Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskYJaGNk/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskYJaGNk/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/security.properties
2025-06-25 08:04:15: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskYJaGNk/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskYJaGNk/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250625 08:04:25.735281 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.126:36875
--webserver_interface=127.1.19.126
--webserver_port=0
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 08:04:25.839641 1345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:25.839900 1345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:25.839970 1345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:25.843092 1345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:25.843158 1345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:25.843189 1345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:25.843222 1345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:25.846908 1345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.126:36875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.19.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:25.848026 1345 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:25.849170 1345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:25.855780 1350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:25.855793 1351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:25.855810 1353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:25.855970 1345 server_base.cc:1048] running on GCE node
I20250625 08:04:25.856595 1345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:25.857481 1345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:25.858631 1345 hybrid_clock.cc:648] HybridClock initialized: now 1750838665858608 us; error 34 us; skew 500 ppm
I20250625 08:04:25.866704 1345 webserver.cc:469] Webserver started at http://127.1.19.126:32959/ using document root <none> and password file <none>
I20250625 08:04:25.867337 1345 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:25.867395 1345 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:25.867599 1345 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 08:04:25.869264 1345 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/instance:
uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
format_stamp: "Formatted at 2025-06-25 08:04:25 on dist-test-slave-4qtp"
I20250625 08:04:25.869741 1345 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal/instance:
uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
format_stamp: "Formatted at 2025-06-25 08:04:25 on dist-test-slave-4qtp"
I20250625 08:04:25.873219 1345 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20250625 08:04:25.875402 1359 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:25.876498 1345 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250625 08:04:25.876614 1345 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
format_stamp: "Formatted at 2025-06-25 08:04:25 on dist-test-slave-4qtp"
I20250625 08:04:25.876711 1345 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:25.935755 1345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:25.936480 1345 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:25.936657 1345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:25.943956 1411 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.126:36875 every 8 connection(s)
I20250625 08:04:25.943953 1345 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.126:36875
I20250625 08:04:25.945022 1345 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
I20250625 08:04:25.947642 1412 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:25.951050 1412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } has no permanent_uuid. Determining permanent_uuid...
I20250625 08:04:25.951717 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 1345
I20250625 08:04:25.951856 1101 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal/instance
I20250625 08:04:25.953402 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.125:38317
--webserver_interface=127.1.19.125
--webserver_port=0
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 08:04:25.962352 1412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } has no permanent_uuid. Determining permanent_uuid...
W20250625 08:04:25.963205 1360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:25.964113 1412 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:26.013353 1412 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } attempt: 1
W20250625 08:04:26.014848 1412 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:26.059064 1415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:26.059299 1415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:26.059370 1415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:26.062537 1415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:26.062604 1415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:26.062637 1415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:26.062669 1415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:26.066359 1415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.125:38317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.19.125
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:26.067505 1415 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:26.068661 1415 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:26.075197 1424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:26.075182 1421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:26.075182 1422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:26.075497 1415 server_base.cc:1048] running on GCE node
I20250625 08:04:26.075991 1415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:26.076931 1415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:26.078080 1415 hybrid_clock.cc:648] HybridClock initialized: now 1750838666078070 us; error 24 us; skew 500 ppm
I20250625 08:04:26.079954 1415 webserver.cc:469] Webserver started at http://127.1.19.125:39497/ using document root <none> and password file <none>
I20250625 08:04:26.080528 1415 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:26.080585 1415 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:26.080766 1415 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 08:04:26.082405 1415 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/instance:
uuid: "0404561252194ce29bf2924fd5e67e22"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.082930 1415 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal/instance:
uuid: "0404561252194ce29bf2924fd5e67e22"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.083024 1412 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } attempt: 2
W20250625 08:04:26.084407 1412 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:26.086669 1415 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250625 08:04:26.088913 1430 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:26.089879 1415 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:26.089988 1415 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
uuid: "0404561252194ce29bf2924fd5e67e22"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.090083 1415 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:26.109220 1415 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:26.110023 1415 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:26.110239 1415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:26.119699 1415 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.125:38317
I20250625 08:04:26.119712 1482 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.125:38317 every 8 connection(s)
I20250625 08:04:26.120755 1415 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
I20250625 08:04:26.123329 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:26.126621 1483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } has no permanent_uuid. Determining permanent_uuid...
I20250625 08:04:26.129555 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 1415
I20250625 08:04:26.129659 1101 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal/instance
I20250625 08:04:26.130293 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.124:37745
--webserver_interface=127.1.19.124
--webserver_port=0
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 08:04:26.136075 1483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } has no permanent_uuid. Determining permanent_uuid...
I20250625 08:04:26.141789 1483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } has no permanent_uuid. Determining permanent_uuid...
W20250625 08:04:26.142752 1432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:26.144258 1483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.19.124:37745: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:26.175580 1412 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } attempt: 3
I20250625 08:04:26.179891 1412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } has no permanent_uuid. Determining permanent_uuid...
W20250625 08:04:26.181075 1412 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.19.124:37745: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:26.193543 1483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } attempt: 1
W20250625 08:04:26.194932 1483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.19.124:37745: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:26.212244 1412 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } attempt: 1
W20250625 08:04:26.213609 1412 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.19.124:37745: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:26.236680 1485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:26.236935 1485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:26.237003 1485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:26.240365 1485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:26.240442 1485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:26.240478 1485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:26.240510 1485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:26.244285 1485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.124:37745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.19.124
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:26.245443 1485 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:26.246613 1485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:26.253073 1493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:26.253069 1492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:26.253078 1495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:26.253324 1485 server_base.cc:1048] running on GCE node
I20250625 08:04:26.253870 1485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:26.254817 1485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:26.256011 1485 hybrid_clock.cc:648] HybridClock initialized: now 1750838666255984 us; error 39 us; skew 500 ppm
I20250625 08:04:26.257839 1485 webserver.cc:469] Webserver started at http://127.1.19.124:44175/ using document root <none> and password file <none>
I20250625 08:04:26.258378 1485 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:26.258428 1485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:26.258652 1485 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 08:04:26.260164 1485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/instance:
uuid: "755093bec4b54a9588598f814c3708e1"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.260578 1485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal/instance:
uuid: "755093bec4b54a9588598f814c3708e1"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.263095 1483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } attempt: 2
I20250625 08:04:26.263911 1485 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
W20250625 08:04:26.264294 1483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.19.124:37745: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:26.266094 1501 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:26.267151 1485 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:26.267266 1485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
uuid: "755093bec4b54a9588598f814c3708e1"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.267352 1485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:26.279419 1485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:26.280064 1485 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:26.280230 1485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:26.287155 1485 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.124:37745
I20250625 08:04:26.287171 1553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.124:37745 every 8 connection(s)
I20250625 08:04:26.288182 1485 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
I20250625 08:04:26.288760 1412 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } attempt: 2
I20250625 08:04:26.291038 1554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:26.294737 1554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } has no permanent_uuid. Determining permanent_uuid...
I20250625 08:04:26.295648 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 1485
I20250625 08:04:26.295754 1101 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal/instance
I20250625 08:04:26.300925 1412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap starting.
I20250625 08:04:26.301877 1554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } has no permanent_uuid. Determining permanent_uuid...
I20250625 08:04:26.303498 1412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:26.304476 1412 log.cc:826] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:26.306174 1554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } has no permanent_uuid. Determining permanent_uuid...
I20250625 08:04:26.306625 1412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: No bootstrap required, opened a new log
I20250625 08:04:26.310882 1412 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:26.311120 1412 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:26.311214 1412 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0b114bd68344b0bd2ee322bd5eccdc, State: Initialized, Role: FOLLOWER
I20250625 08:04:26.311780 1412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:26.312638 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap starting.
I20250625 08:04:26.312943 1412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:26.315402 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:26.317232 1554 log.cc:826] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:26.318912 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: No bootstrap required, opened a new log
I20250625 08:04:26.319203 1345 master_runner.cc:186] Error getting master registration for 127.1.19.125:38317: OK, instance_id { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" instance_seqno: 1750838666117503 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250625 08:04:26.321527 1345 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:26.321918 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.322167 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:26.322391 1554 raft_consensus.cc:357] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:26.322693 1554 raft_consensus.cc:383] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:26.322796 1554 raft_consensus.cc:738] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755093bec4b54a9588598f814c3708e1, State: Initialized, Role: FOLLOWER
I20250625 08:04:26.323344 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:26.323751 1581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.323904 1581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:26.324445 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: configured and running, proceeding with master startup.
W20250625 08:04:26.324777 1577 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 08:04:26.324916 1577 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250625 08:04:26.331403 1485 master_runner.cc:186] Error getting master registration for 127.1.19.125:38317: OK, instance_id { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" instance_seqno: 1750838666117503 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250625 08:04:26.333257 1485 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:04:26.340601 1595 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 08:04:26.340776 1595 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250625 08:04:26.355451 1483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } attempt: 3
I20250625 08:04:26.365991 1483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap starting.
I20250625 08:04:26.368927 1483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:26.370102 1483 log.cc:826] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:26.384495 1483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: No bootstrap required, opened a new log
I20250625 08:04:26.388676 1483 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:26.389011 1483 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:26.389123 1483 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0404561252194ce29bf2924fd5e67e22, State: Initialized, Role: FOLLOWER
I20250625 08:04:26.389719 1483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:26.391242 1483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:26.395823 1599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.396097 1599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:26.403002 1415 master_runner.cc:263] No leader master found from master 0404561252194ce29bf2924fd5e67e22
I20250625 08:04:26.403100 1415 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:04:26.411204 1613 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 08:04:26.414398 1613 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250625 08:04:26.438289 1365 tablet.cc:2378] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 08:04:26.531456 1507 tablet.cc:2378] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 08:04:26.557202 1581 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:26.557425 1581 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:26.559286 1386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
I20250625 08:04:26.559614 1386 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 0.
I20250625 08:04:26.560204 1505 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters:
I20250625 08:04:26.560739 1581 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
I20250625 08:04:26.560842 1581 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:26.560895 1581 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:26.560933 1581 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:26.562075 1581 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:26.563074 1453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0404561252194ce29bf2924fd5e67e22" is_pre_election: true
I20250625 08:04:26.563318 1386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
I20250625 08:04:26.563391 1453 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 0.
I20250625 08:04:26.563431 1386 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:26.564502 1386 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 1.
I20250625 08:04:26.565237 1505 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters:
I20250625 08:04:26.565449 1581 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
I20250625 08:04:26.565557 1581 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:26.565616 1581 raft_consensus.cc:695] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 1 LEADER]: Becoming Leader. State: Replica: 755093bec4b54a9588598f814c3708e1, State: Running, Role: LEADER
I20250625 08:04:26.565927 1581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:26.566939 1453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0404561252194ce29bf2924fd5e67e22"
I20250625 08:04:26.567077 1453 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:26.568233 1453 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 1.
I20250625 08:04:26.570205 1581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 1 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.570425 1581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:26.575037 1618 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:26.576141 1618 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:26.582005 1453 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 1 FOLLOWER]: Refusing update from remote peer 755093bec4b54a9588598f814c3708e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 08:04:26.582804 1616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:26.587388 1599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 1 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.587548 1599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:26.588799 1386 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 1 FOLLOWER]: Refusing update from remote peer 755093bec4b54a9588598f814c3708e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 08:04:26.589530 1616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:26.590793 1581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.590907 1581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:26.591984 1599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.592103 1599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:26.592594 1619 mvcc.cc:204] Tried to move back new op lower bound from 7171435178311172096 to 7171435178266980352. Current Snapshot: MvccSnapshot[applied={T|T < 7171435178311172096}]
I20250625 08:04:26.594170 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 1 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.594307 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:26.594398 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.594483 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:26.597608 1620 mvcc.cc:204] Tried to move back new op lower bound from 7171435178311172096 to 7171435178266980352. Current Snapshot: MvccSnapshot[applied={T|T < 7171435178311172096}]
I20250625 08:04:26.598346 1618 catalog_manager.cc:1349] Generated new cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:26.598467 1618 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:26.600440 1624 mvcc.cc:204] Tried to move back new op lower bound from 7171435178311172096 to 7171435178266980352. Current Snapshot: MvccSnapshot[applied={T|T < 7171435178311172096}]
I20250625 08:04:26.611277 1616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:26.611429 1616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:26.615655 1618 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 08:04:26.616672 1618 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:26.628022 1618 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Generated new TSK 0
I20250625 08:04:26.629112 1618 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:26.629971 1618 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:26.630426 1618 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250625 08:04:26.647300 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.65:0
--local_ip_for_outbound_sockets=127.1.19.65
--webserver_interface=127.1.19.65
--webserver_port=0
--tserver_master_addrs=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250625 08:04:26.757876 1629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:26.758126 1629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:26.758179 1629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:26.761466 1629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:26.761605 1629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.65
I20250625 08:04:26.765534 1629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.19.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--never_fsync=true
--heap_profile_path=/tmp/kudu.1629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.65
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:26.766868 1629 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:26.768016 1629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:26.775797 1635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:26.775810 1634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:26.775942 1629 server_base.cc:1048] running on GCE node
W20250625 08:04:26.775835 1637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:26.776417 1629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:26.776973 1629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:26.778129 1629 hybrid_clock.cc:648] HybridClock initialized: now 1750838666778118 us; error 28 us; skew 500 ppm
I20250625 08:04:26.779965 1629 webserver.cc:469] Webserver started at http://127.1.19.65:43789/ using document root <none> and password file <none>
I20250625 08:04:26.780545 1629 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:26.780609 1629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:26.780860 1629 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 08:04:26.782603 1629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/data/instance:
uuid: "eebfa8472fe04080ac012e2c91bedaae"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.783102 1629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/instance:
uuid: "eebfa8472fe04080ac012e2c91bedaae"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.786557 1629 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250625 08:04:26.788825 1643 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:26.789803 1629 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:26.789924 1629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal
uuid: "eebfa8472fe04080ac012e2c91bedaae"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.790017 1629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:26.805948 1629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:26.806694 1629 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:26.806883 1629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:26.807472 1629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 08:04:26.808512 1629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 08:04:26.808565 1629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:26.808633 1629 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 08:04:26.808663 1629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:26.818043 1629 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.65:46085
I20250625 08:04:26.818084 1756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.65:46085 every 8 connection(s)
I20250625 08:04:26.819108 1629 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/data/info.pb
I20250625 08:04:26.823601 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 1629
I20250625 08:04:26.823717 1101 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/instance
I20250625 08:04:26.826259 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.66:0
--local_ip_for_outbound_sockets=127.1.19.66
--webserver_interface=127.1.19.66
--webserver_port=0
--tserver_master_addrs=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 08:04:26.830610 1757 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:26.830649 1759 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:26.830657 1758 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:26.830914 1758 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:26.830914 1757 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:26.830914 1759 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:26.831645 1759 heartbeater.cc:507] Master 127.1.19.125:38317 requested a full tablet report, sending...
I20250625 08:04:26.831645 1757 heartbeater.cc:507] Master 127.1.19.124:37745 requested a full tablet report, sending...
I20250625 08:04:26.831645 1758 heartbeater.cc:507] Master 127.1.19.126:36875 requested a full tablet report, sending...
I20250625 08:04:26.833230 1376 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:26.833302 1518 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:26.835078 1443 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:26.835119 1518 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.65:40635
W20250625 08:04:26.952690 1766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:26.952926 1766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:26.952986 1766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:26.956821 1766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:26.957017 1766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.66
I20250625 08:04:26.961447 1766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.19.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--never_fsync=true
--heap_profile_path=/tmp/kudu.1766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.66
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:26.962698 1766 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:26.963910 1766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:26.972184 1774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:26.972199 1771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:26.972368 1772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:26.972654 1766 server_base.cc:1048] running on GCE node
I20250625 08:04:26.973150 1766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:26.973781 1766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:26.974951 1766 hybrid_clock.cc:648] HybridClock initialized: now 1750838666974925 us; error 44 us; skew 500 ppm
I20250625 08:04:26.976980 1766 webserver.cc:469] Webserver started at http://127.1.19.66:35789/ using document root <none> and password file <none>
I20250625 08:04:26.977574 1766 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:26.977634 1766 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:26.977840 1766 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 08:04:26.979533 1766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/data/instance:
uuid: "de62034f80d9463dbd6386fc3e90fdaf"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.980022 1766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/instance:
uuid: "de62034f80d9463dbd6386fc3e90fdaf"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.983645 1766 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250625 08:04:26.985970 1780 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:26.987144 1766 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250625 08:04:26.987246 1766 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal
uuid: "de62034f80d9463dbd6386fc3e90fdaf"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:26.987357 1766 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:27.008287 1766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:27.008991 1766 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:27.009184 1766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:27.009760 1766 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 08:04:27.010793 1766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 08:04:27.010843 1766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:27.010886 1766 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 08:04:27.010906 1766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:27.020342 1766 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.66:46071
I20250625 08:04:27.020378 1893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.66:46071 every 8 connection(s)
I20250625 08:04:27.021410 1766 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/data/info.pb
I20250625 08:04:27.025378 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 1766
I20250625 08:04:27.025522 1101 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/instance
I20250625 08:04:27.029287 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.67:0
--local_ip_for_outbound_sockets=127.1.19.67
--webserver_interface=127.1.19.67
--webserver_port=0
--tserver_master_addrs=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 08:04:27.033028 1894 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:27.033046 1895 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:27.033454 1895 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:27.033454 1894 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:27.033955 1896 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:27.034032 1896 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:27.034257 1895 heartbeater.cc:507] Master 127.1.19.126:36875 requested a full tablet report, sending...
I20250625 08:04:27.034287 1894 heartbeater.cc:507] Master 127.1.19.124:37745 requested a full tablet report, sending...
I20250625 08:04:27.034257 1896 heartbeater.cc:507] Master 127.1.19.125:38317 requested a full tablet report, sending...
I20250625 08:04:27.035496 1518 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:27.035594 1443 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:27.035619 1376 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:27.036247 1518 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.66:47231
W20250625 08:04:27.148811 1901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:27.149168 1901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:27.149274 1901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:27.152819 1901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:27.152961 1901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.67
I20250625 08:04:27.157006 1901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.19.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--never_fsync=true
--heap_profile_path=/tmp/kudu.1901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.67
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:27.158763 1901 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:27.159951 1901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:27.168208 1907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:27.168201 1909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:27.168359 1901 server_base.cc:1048] running on GCE node
W20250625 08:04:27.168514 1906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:27.168927 1901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:27.169586 1901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:27.170743 1901 hybrid_clock.cc:648] HybridClock initialized: now 1750838667170719 us; error 41 us; skew 500 ppm
I20250625 08:04:27.172863 1901 webserver.cc:469] Webserver started at http://127.1.19.67:32869/ using document root <none> and password file <none>
I20250625 08:04:27.173681 1901 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:27.173807 1901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:27.174038 1901 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 08:04:27.175933 1901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/data/instance:
uuid: "5dae8595b9484b57bc7ccea543f83130"
format_stamp: "Formatted at 2025-06-25 08:04:27 on dist-test-slave-4qtp"
I20250625 08:04:27.176498 1901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/instance:
uuid: "5dae8595b9484b57bc7ccea543f83130"
format_stamp: "Formatted at 2025-06-25 08:04:27 on dist-test-slave-4qtp"
I20250625 08:04:27.180541 1901 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250625 08:04:27.183197 1915 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:27.184376 1901 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250625 08:04:27.184499 1901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal
uuid: "5dae8595b9484b57bc7ccea543f83130"
format_stamp: "Formatted at 2025-06-25 08:04:27 on dist-test-slave-4qtp"
I20250625 08:04:27.184595 1901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:27.196322 1901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:27.197068 1901 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:27.197232 1901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:27.197775 1901 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 08:04:27.198800 1901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 08:04:27.198851 1901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:27.198892 1901 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 08:04:27.198908 1901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:27.208035 1901 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.67:37005
I20250625 08:04:27.208063 2028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.67:37005 every 8 connection(s)
I20250625 08:04:27.209091 1901 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/data/info.pb
I20250625 08:04:27.217227 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 1901
I20250625 08:04:27.217362 1101 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/instance
I20250625 08:04:27.219478 2030 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:27.219760 2030 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:27.220449 2030 heartbeater.cc:507] Master 127.1.19.126:36875 requested a full tablet report, sending...
I20250625 08:04:27.220980 2029 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:27.221060 2029 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:27.221268 2029 heartbeater.cc:507] Master 127.1.19.124:37745 requested a full tablet report, sending...
I20250625 08:04:27.221457 2031 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:27.221521 2031 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:27.221669 2031 heartbeater.cc:507] Master 127.1.19.125:38317 requested a full tablet report, sending...
I20250625 08:04:27.221714 1376 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:27.221971 1517 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:27.222697 1517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.67:55199
I20250625 08:04:27.223342 1443 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:27.237154 1101 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 08:04:27.254387 1517 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57610:
name: "default.Table_31023e55a4384ac88f8415871e52d142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.254523 1511 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57610:
name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.254674 1516 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57610:
name: "default.Table_d28f89eb30604a7abf867f68131b2c1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.254693 1513 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57610:
name: "default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.254959 1510 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57610:
name: "default.Table_c1ad5f1d55214059b710e6acaa0f61d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.255012 1514 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57610:
name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.255254 1512 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57610:
name: "default.Table_b49cbf7ee55042b39d7b5d3946a35618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.255481 1509 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57610:
name: "default.Table_98189dd9ad454e5d8ee6edd848927c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.254343 1515 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57610:
name: "default.Table_a3a8c500599e482da5b7120ec26a797a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.255911 1101 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu with pid 1485
W20250625 08:04:27.265524 2037 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.124:37745)
I20250625 08:04:27.326252 1577 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:27.326349 1577 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: loading cluster ID for follower catalog manager: success
I20250625 08:04:27.328195 1577 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: acquiring CA information for follower catalog manager: success
I20250625 08:04:27.329449 1577 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:27.337137 1757 heartbeater.cc:499] Master 127.1.19.124:37745 was elected leader, sending a full tablet report...
W20250625 08:04:27.338070 1757 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:27.415969 1613 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:27.416054 1613 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: loading cluster ID for follower catalog manager: success
I20250625 08:04:27.418048 1613 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: acquiring CA information for follower catalog manager: success
I20250625 08:04:27.419400 1613 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:27.533795 2076 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:27.533943 2076 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:27.535020 2076 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:27.535454 1453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0404561252194ce29bf2924fd5e67e22" is_pre_election: true
W20250625 08:04:27.536257 1361 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:27.536339 1361 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:27.536540 2076 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:27.537972 1894 heartbeater.cc:499] Master 127.1.19.124:37745 was elected leader, sending a full tablet report...
W20250625 08:04:27.538900 1894 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:27.594345 2080 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:27.594537 2080 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:27.595501 2080 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:27.595995 1386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
I20250625 08:04:27.596187 1386 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0404561252194ce29bf2924fd5e67e22 in term 1.
I20250625 08:04:27.596593 1434 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters:
I20250625 08:04:27.596870 2080 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 08:04:27.596938 2080 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:27.597034 2080 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 1 FOLLOWER]: Advancing to term 2
W20250625 08:04:27.597371 1432 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:27.598137 2080 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:27.598526 2080 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 2 election: Requested vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:27.598865 1386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
I20250625 08:04:27.598994 1386 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 1 FOLLOWER]: Advancing to term 2
W20250625 08:04:27.599520 1432 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:27.600036 1386 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0404561252194ce29bf2924fd5e67e22 in term 2.
I20250625 08:04:27.600368 1434 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 755093bec4b54a9588598f814c3708e1
I20250625 08:04:27.600550 2080 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 08:04:27.600735 2080 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 2 LEADER]: Becoming Leader. State: Replica: 0404561252194ce29bf2924fd5e67e22, State: Running, Role: LEADER
I20250625 08:04:27.601037 2080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:27.602034 2083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0404561252194ce29bf2924fd5e67e22. Latest consensus state: current_term: 2 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:27.602162 2083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:27.602546 2084 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:27.603389 2084 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:27.603760 2084 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:27.603808 2084 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:27.604358 2084 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:27.604727 2084 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Loaded TSK: 0
I20250625 08:04:27.605489 2084 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:27.605882 2084 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:27.606187 2084 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250625 08:04:27.626742 1447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.626693 1446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_b49cbf7ee55042b39d7b5d3946a35618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.626910 1442 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_d28f89eb30604a7abf867f68131b2c1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.626896 1440 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.626927 1438 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.626487 1439 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_c1ad5f1d55214059b710e6acaa0f61d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.627013 1441 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_a3a8c500599e482da5b7120ec26a797a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.626420 1443 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_98189dd9ad454e5d8ee6edd848927c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.626528 1445 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_d78071777f3646548f7f75cea674ecdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:27.626581 1444 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_31023e55a4384ac88f8415871e52d142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 08:04:27.654700 1442 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d28f89eb30604a7abf867f68131b2c1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:04:27.724582 2029 heartbeater.cc:499] Master 127.1.19.124:37745 was elected leader, sending a full tablet report...
W20250625 08:04:27.726946 2029 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:27.856212 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.124:37745
--webserver_interface=127.1.19.124
--webserver_port=44175
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 08:04:28.027391 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:28.027725 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:28.027813 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:28.033004 2087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:28.033110 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:28.033141 2087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:28.033167 2087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:28.039371 2087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.124:37745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.19.124
--webserver_port=44175
--never_fsync=true
--heap_profile_path=/tmp/kudu.2087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:28.041527 2087 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:28.043156 2087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:28.046295 1432 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 -> Peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Couldn't send request to peer 755093bec4b54a9588598f814c3708e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250625 08:04:28.052033 2093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:28.052033 2094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:28.052592 2096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:28.052852 2087 server_base.cc:1048] running on GCE node
I20250625 08:04:28.053424 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:28.054558 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:28.055725 2087 hybrid_clock.cc:648] HybridClock initialized: now 1750838668055715 us; error 38 us; skew 500 ppm
I20250625 08:04:28.058318 2087 webserver.cc:469] Webserver started at http://127.1.19.124:44175/ using document root <none> and password file <none>
I20250625 08:04:28.059046 2087 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:28.059136 2087 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:28.063583 2087 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250625 08:04:28.066320 2102 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:28.067927 2087 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250625 08:04:28.068221 2087 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
uuid: "755093bec4b54a9588598f814c3708e1"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:28.068957 2087 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:28.100360 2087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:28.101406 2087 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:28.101737 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:28.111475 2154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.124:37745 every 8 connection(s)
I20250625 08:04:28.111470 2087 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.124:37745
I20250625 08:04:28.113195 2087 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
I20250625 08:04:28.115646 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 2087
I20250625 08:04:28.122020 2155 sys_catalog.cc:263] Verifying existing consensus state
I20250625 08:04:28.124264 2155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap starting.
I20250625 08:04:28.127511 1386 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 2 FOLLOWER]: Refusing update from remote peer 0404561252194ce29bf2924fd5e67e22: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250625 08:04:28.128281 2083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250625 08:04:28.129899 2159 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:28.130120 2159 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.131261 1453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0404561252194ce29bf2924fd5e67e22" is_pre_election: true
I20250625 08:04:28.132889 2159 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:28.133034 2159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0404561252194ce29bf2924fd5e67e22. Latest consensus state: current_term: 2 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.133044 2083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.133359 2083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:28.133354 2159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:28.133579 2159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.133668 2159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:28.144573 2155 log.cc:826] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:28.152604 2155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:28.153112 2155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap complete.
I20250625 08:04:28.156834 2155 raft_consensus.cc:357] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.157330 2155 raft_consensus.cc:738] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755093bec4b54a9588598f814c3708e1, State: Initialized, Role: FOLLOWER
I20250625 08:04:28.158002 2155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.159439 2155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:28.159487 2129 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755093bec4b54a9588598f814c3708e1" is_pre_election: true
I20250625 08:04:28.159847 2129 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f0b114bd68344b0bd2ee322bd5eccdc in term 1.
I20250625 08:04:28.160534 1361 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22
I20250625 08:04:28.160775 2163 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250625 08:04:28.160853 2163 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:28.160904 2163 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 2 FOLLOWER]: Advancing to term 3
I20250625 08:04:28.162132 2163 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.162631 2163 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 3 election: Requested vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:28.163270 2129 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755093bec4b54a9588598f814c3708e1"
I20250625 08:04:28.163270 1453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0404561252194ce29bf2924fd5e67e22"
I20250625 08:04:28.163399 2129 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 1 FOLLOWER]: Advancing to term 3
I20250625 08:04:28.165964 2129 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f0b114bd68344b0bd2ee322bd5eccdc in term 3.
I20250625 08:04:28.166399 1361 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22
I20250625 08:04:28.166790 2163 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 3 FOLLOWER]: Leader election won for term 3
I20250625 08:04:28.166883 2163 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 3 LEADER]: Becoming Leader. State: Replica: 5f0b114bd68344b0bd2ee322bd5eccdc, State: Running, Role: LEADER
I20250625 08:04:28.167209 2163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.168202 2163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0b114bd68344b0bd2ee322bd5eccdc. Latest consensus state: current_term: 3 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.168313 2163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:28.168874 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.169088 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:28.169373 2184 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:28.171579 2184 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:28.172057 2184 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:28.172160 2184 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:28.172852 2184 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:28.173058 1442 catalog_manager.cc:2149] created HMS catalog entry for table [id=93ac3d33131249dd809793b74ef4523f]
I20250625 08:04:28.173393 2184 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Loaded TSK: 0
I20250625 08:04:28.174243 2183 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:28.174299 2183 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: loading cluster ID for follower catalog manager: success
I20250625 08:04:28.176244 1386 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 3 LEADER]: Rejecting Update request from peer 0404561252194ce29bf2924fd5e67e22 for earlier term 2. Current term is 3. Ops: [2.6-2.6]
I20250625 08:04:28.176853 2183 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: acquiring CA information for follower catalog manager: success
I20250625 08:04:28.176895 2083 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250625 08:04:28.177052 2083 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Peer 5f0b114bd68344b0bd2ee322bd5eccdc log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250625 08:04:28.177122 2083 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 2 LEADER]: Stepping down as leader of term 2
I20250625 08:04:28.177156 2083 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0404561252194ce29bf2924fd5e67e22, State: Running, Role: LEADER
I20250625 08:04:28.177323 2083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.177593 2083 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 2 FOLLOWER]: Advancing to term 3
I20250625 08:04:28.178601 2183 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250625 08:04:28.178665 1439 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c1ad5f1d55214059b710e6acaa0f61d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 08:04:28.178987 1446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b49cbf7ee55042b39d7b5d3946a35618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 08:04:28.179400 1441 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a3a8c500599e482da5b7120ec26a797a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 08:04:28.179576 1440 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c06b3617aaf24ba4b0ae0b3926081026 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 08:04:28.180027 1447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b0a79766b44a4a6882c263b6c2c46ed2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 08:04:28.180224 1444 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_31023e55a4384ac88f8415871e52d142 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 08:04:28.180856 1445 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d78071777f3646548f7f75cea674ecdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 08:04:28.181492 1443 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_98189dd9ad454e5d8ee6edd848927c15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 08:04:28.182113 1438 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ba2c9866b9f44ecd85b09bccf0e63aa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:04:28.185225 2184 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:28.186508 2184 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:28.186983 2184 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250625 08:04:28.235703 1439 catalog_manager.cc:2149] created HMS catalog entry for table [id=b1d0024d01cf45749d4d302a0d2dde73]
W20250625 08:04:28.236518 1439 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER
I20250625 08:04:28.237841 2029 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:28.238005 2029 heartbeater.cc:499] Master 127.1.19.124:37745 was elected leader, sending a full tablet report...
I20250625 08:04:28.239161 2110 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" instance_seqno: 1750838667205184) as {username='slave'} at 127.1.19.67:51219; Asking this server to re-register.
I20250625 08:04:28.239720 2029 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:28.240782 2110 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:28.277053 1446 catalog_manager.cc:2149] created HMS catalog entry for table [id=bb802033c2134de9ac6187235037ad4c]
W20250625 08:04:28.277666 1446 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER
I20250625 08:04:28.312373 1441 catalog_manager.cc:2149] created HMS catalog entry for table [id=24b00899d74d48f5866b7d034091f802]
W20250625 08:04:28.313055 1441 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER
I20250625 08:04:28.326682 1453 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 3 FOLLOWER]: Refusing update from remote peer 5f0b114bd68344b0bd2ee322bd5eccdc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20250625 08:04:28.327333 2163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250625 08:04:28.327486 2163 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Peer 0404561252194ce29bf2924fd5e67e22 log is divergent from this leader: its last log entry 2.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250625 08:04:28.328202 2129 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 3 FOLLOWER]: Refusing update from remote peer 5f0b114bd68344b0bd2ee322bd5eccdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20250625 08:04:28.328786 2163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250625 08:04:28.329200 1453 pending_rounds.cc:85] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Aborting all ops after (but not including) 5
I20250625 08:04:28.329299 1453 pending_rounds.cc:107] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Aborting uncommitted WRITE_OP operation due to leader change: 2.6
W20250625 08:04:28.330140 1442 catalog_manager.cc:2177] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20250625 08:04:28.330336 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0b114bd68344b0bd2ee322bd5eccdc. Latest consensus state: current_term: 3 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.330430 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:28.332657 2163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.332665 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0b114bd68344b0bd2ee322bd5eccdc. Latest consensus state: current_term: 3 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.332770 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:28.332772 2163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:28.332880 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.332952 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:28.334148 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.334257 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:28.341836 2158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.342020 2158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:28.343349 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.343492 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:28.345775 2191 mvcc.cc:204] Tried to move back new op lower bound from 7171435185455648768 to 7171435182496149504. Current Snapshot: MvccSnapshot[applied={T|T < 7171435185455648768 or (T in {7171435185455648768})}]
I20250625 08:04:28.351994 1757 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:28.352187 1757 heartbeater.cc:499] Master 127.1.19.124:37745 was elected leader, sending a full tablet report...
I20250625 08:04:28.353009 2110 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" instance_seqno: 1750838666815429) as {username='slave'} at 127.1.19.65:57645; Asking this server to re-register.
I20250625 08:04:28.353575 1757 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:28.354430 2110 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:28.354705 1440 catalog_manager.cc:2149] created HMS catalog entry for table [id=91897f72c011456fb9b3fa2abb08b6b1]
W20250625 08:04:28.355396 1440 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER
I20250625 08:04:28.402478 1447 catalog_manager.cc:2149] created HMS catalog entry for table [id=76655d21216448fb8307987100f8513f]
W20250625 08:04:28.403112 1447 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER
I20250625 08:04:28.434468 1444 catalog_manager.cc:2149] created HMS catalog entry for table [id=7825ea67af2d42b68200097030b08116]
W20250625 08:04:28.435122 1444 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER
I20250625 08:04:28.459235 1445 catalog_manager.cc:2149] created HMS catalog entry for table [id=0e43d85f40e545d9b31c5cb3e62f89a8]
W20250625 08:04:28.459861 1445 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER
I20250625 08:04:28.481835 1443 catalog_manager.cc:2149] created HMS catalog entry for table [id=d2b3f7fc195b480f9e7d191a8ed99599]
W20250625 08:04:28.482522 1443 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER
I20250625 08:04:28.504873 1438 catalog_manager.cc:2149] created HMS catalog entry for table [id=b53c21c44e124643927d73a514201a8e]
W20250625 08:04:28.505528 1438 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER
I20250625 08:04:28.546487 1894 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:28.546670 1894 heartbeater.cc:499] Master 127.1.19.124:37745 was elected leader, sending a full tablet report...
I20250625 08:04:28.547410 2110 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" instance_seqno: 1750838667017582) as {username='slave'} at 127.1.19.66:52893; Asking this server to re-register.
I20250625 08:04:28.547966 1894 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:28.548862 2110 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:28.729810 2030 heartbeater.cc:499] Master 127.1.19.126:36875 was elected leader, sending a full tablet report...
I20250625 08:04:28.834281 2161 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:28.834487 2161 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.835027 2161 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:28.836301 1386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
I20250625 08:04:28.839008 1758 heartbeater.cc:499] Master 127.1.19.126:36875 was elected leader, sending a full tablet report...
I20250625 08:04:28.842209 2129 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "755093bec4b54a9588598f814c3708e1" is_pre_election: true
I20250625 08:04:28.842741 1432 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:28.843070 2161 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250625 08:04:28.843063 2193 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:28.843165 2193 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.844667 2193 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
I20250625 08:04:28.845214 1453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "0404561252194ce29bf2924fd5e67e22" is_pre_election: true
I20250625 08:04:28.845366 1453 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 3.
I20250625 08:04:28.846148 1386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
I20250625 08:04:28.846844 2103 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1; no voters:
I20250625 08:04:28.847481 2193 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250625 08:04:28.847545 2193 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:28.847589 2193 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 3 FOLLOWER]: Advancing to term 4
I20250625 08:04:28.848706 2193 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.849125 2193 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 4 election: Requested vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
I20250625 08:04:28.849747 1386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
I20250625 08:04:28.850806 1453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "0404561252194ce29bf2924fd5e67e22"
I20250625 08:04:28.850956 1453 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 3 FOLLOWER]: Advancing to term 4
I20250625 08:04:28.852165 1453 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 4.
I20250625 08:04:28.852726 2103 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc
I20250625 08:04:28.853483 2193 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 4 FOLLOWER]: Leader election won for term 4
I20250625 08:04:28.853701 2193 raft_consensus.cc:695] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 4 LEADER]: Becoming Leader. State: Replica: 755093bec4b54a9588598f814c3708e1, State: Running, Role: LEADER
I20250625 08:04:28.854235 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.856261 2200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 4 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:28.856607 2200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:28.857612 2201 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:28.858994 2201 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:28.859470 2201 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:28.859519 2201 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:28.860195 2201 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:28.860692 2201 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Loaded TSK: 0
I20250625 08:04:28.861732 2201 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:28.862267 2201 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:28.862772 2201 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 4
I20250625 08:04:28.867605 2129 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 4 LEADER]: Rejecting Update request from peer 5f0b114bd68344b0bd2ee322bd5eccdc for earlier term 3. Current term is 4. Ops: []
I20250625 08:04:28.868968 2202 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 }, Status: INVALID_TERM, Last received: 3.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250625 08:04:28.869277 2202 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Peer 755093bec4b54a9588598f814c3708e1 log is divergent from this leader: its last log entry 4.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250625 08:04:28.869364 2202 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 3 LEADER]: Stepping down as leader of term 3
I20250625 08:04:28.869406 2202 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f0b114bd68344b0bd2ee322bd5eccdc, State: Running, Role: LEADER
I20250625 08:04:28.869644 2202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:28.869993 2202 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 3 FOLLOWER]: Advancing to term 4
I20250625 08:04:29.028822 1439 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=b1d0024d01cf45749d4d302a0d2dde73]
I20250625 08:04:29.029407 1439 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 26) took 1403 ms. Trace:
I20250625 08:04:29.029493 1439 rpcz_store.cc:276] 0625 08:04:27.626189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626273 (+ 84us) service_pool.cc:225] Handling call
0625 08:04:28.178769 (+552496us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:28.179089 (+ 320us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.235690 (+ 56601us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.236382 (+ 692us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.236385 (+ 3us) write_op.cc:454] Released schema lock
0625 08:04:28.236616 (+ 231us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.029381 (+792765us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.236285 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER)
0625 08:04:28.236297 (+ 12us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.236302 (+ 5us) write_op.cc:454] Released schema lock
0625 08:04:28.236305 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":268174,"Hive Metastore.run_cpu_time_us":497,"Hive Metastore.run_wall_time_us":580337,"spinlock_wait_cycles":46976,"child_traces":[["op",{}]]}
W20250625 08:04:29.030592 1439 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.19.67:50743 (request call id 2) took 1304 ms (client timeout 1000 ms). Trace:
W20250625 08:04:29.030659 1439 rpcz_store.cc:269] 0625 08:04:27.725683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:29.030428 (+1304745us) service_pool.cc:212] Skipping call since client already timed out
0625 08:04:29.030573 (+ 145us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250625 08:04:29.030825 1439 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.19.65:38855 (request call id 3) took 1193 ms (client timeout 1000 ms). Trace:
W20250625 08:04:29.030886 1439 rpcz_store.cc:269] 0625 08:04:27.837365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:29.030795 (+1193430us) service_pool.cc:212] Skipping call since client already timed out
0625 08:04:29.030819 (+ 24us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250625 08:04:29.031147 2036 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.125:38317)
W20250625 08:04:29.031280 1439 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.1.19.66:56967 (request call id 3) took 991 ms (client timeout 1000 ms). Trace:
W20250625 08:04:29.031329 1439 rpcz_store.cc:269] 0625 08:04:28.040107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:29.030991 (+990884us) service_pool.cc:225] Handling call
0625 08:04:29.031268 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 08:04:29.033726 2087 master_runner.cc:232] Existing masters have differing terms: 3 vs 4
I20250625 08:04:29.033850 2087 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:29.038218 2110 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_c1ad5f1d55214059b710e6acaa0f61d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.040768 1895 heartbeater.cc:499] Master 127.1.19.126:36875 was elected leader, sending a full tablet report...
I20250625 08:04:29.046296 2182 catalog_manager.cc:2747] Deleting table default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=b1d0024d01cf45749d4d302a0d2dde73] in response to Hive Metastore notification log event 11
W20250625 08:04:29.046675 2182 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 11 DROP_TABLE default.table_c1ad5f1d55214059b710e6acaa0f61d6: Not found: the table does not exist: table_id: "b1d0024d01cf45749d4d302a0d2dde73" table_name: "default.table_c1ad5f1d55214059b710e6acaa0f61d6"
I20250625 08:04:29.049441 1386 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 4 FOLLOWER]: Refusing update from remote peer 755093bec4b54a9588598f814c3708e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20250625 08:04:29.049468 1453 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Refusing update from remote peer 755093bec4b54a9588598f814c3708e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
I20250625 08:04:29.049954 2193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250625 08:04:29.050115 2200 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250625 08:04:29.051494 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 4 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:29.051517 2203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 4 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:29.051610 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:29.051649 2203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:29.054196 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:29.054353 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:29.054692 2203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:29.054806 2203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:29.055441 2200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:29.055552 2200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:29.056747 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:29.056867 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
W20250625 08:04:29.060297 2110 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c1ad5f1d55214059b710e6acaa0f61d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:04:29.110260 1446 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=bb802033c2134de9ac6187235037ad4c]
I20250625 08:04:29.110607 1446 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 28) took 1484 ms. Trace:
I20250625 08:04:29.110679 1446 rpcz_store.cc:276] 0625 08:04:27.626312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626521 (+ 209us) service_pool.cc:225] Handling call
0625 08:04:28.179057 (+552536us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:28.179304 (+ 247us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.277040 (+ 97736us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.277616 (+ 576us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.277618 (+ 2us) write_op.cc:454] Released schema lock
0625 08:04:28.277732 (+ 114us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.110581 (+832849us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.277564 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER)
0625 08:04:28.277573 (+ 9us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.277579 (+ 6us) write_op.cc:454] Released schema lock
0625 08:04:28.277582 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":807346,"Hive Metastore.run_cpu_time_us":419,"Hive Metastore.run_wall_time_us":122692,"mutex_wait_us":352,"child_traces":[["op",{}]]}
I20250625 08:04:29.118463 2119 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_b49cbf7ee55042b39d7b5d3946a35618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.151773 2110 catalog_manager.cc:2149] created HMS catalog entry for table [id=f060ac216bb54def8fc8a79571936089]
I20250625 08:04:29.163379 2182 catalog_manager.cc:2747] Deleting table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=bb802033c2134de9ac6187235037ad4c] in response to Hive Metastore notification log event 12
W20250625 08:04:29.163581 2182 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 12 DROP_TABLE default.table_b49cbf7ee55042b39d7b5d3946a35618: Not found: the table does not exist: table_id: "bb802033c2134de9ac6187235037ad4c" table_name: "default.table_b49cbf7ee55042b39d7b5d3946a35618"
I20250625 08:04:29.164273 2110 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_c1ad5f1d55214059b710e6acaa0f61d6" } new_table_name: "default.Table_23c54834c68a47bd9654b34281b53e7d" modify_external_catalogs: true
W20250625 08:04:29.170605 2119 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b49cbf7ee55042b39d7b5d3946a35618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:04:29.188935 1828 tablet_service.cc:1468] Processing CreateTablet for tablet cd42120cc1ee43c1bd11944529eee1a5 (DEFAULT_TABLE table=default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.190127 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd42120cc1ee43c1bd11944529eee1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.191951 1691 tablet_service.cc:1468] Processing CreateTablet for tablet cd42120cc1ee43c1bd11944529eee1a5 (DEFAULT_TABLE table=default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.192989 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd42120cc1ee43c1bd11944529eee1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.192924 1963 tablet_service.cc:1468] Processing CreateTablet for tablet cd42120cc1ee43c1bd11944529eee1a5 (DEFAULT_TABLE table=default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.193850 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd42120cc1ee43c1bd11944529eee1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.198032 2221 tablet_bootstrap.cc:492] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.200505 2221 tablet_bootstrap.cc:654] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.201196 2223 tablet_bootstrap.cc:492] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.201617 2221 log.cc:826] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:29.203689 2221 tablet_bootstrap.cc:492] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.203922 2221 ts_tablet_manager.cc:1397] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20250625 08:04:29.207342 2221 raft_consensus.cc:357] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.207600 2221 raft_consensus.cc:383] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.207664 2221 raft_consensus.cc:738] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.208006 1441 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=24b00899d74d48f5866b7d034091f802]
I20250625 08:04:29.208274 1441 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 32) took 1581 ms. Trace:
I20250625 08:04:29.208328 1441 rpcz_store.cc:276] 0625 08:04:27.626692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626946 (+ 254us) service_pool.cc:225] Handling call
0625 08:04:28.179458 (+552512us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:28.179749 (+ 291us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.312358 (+132609us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.313004 (+ 646us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.313007 (+ 3us) write_op.cc:454] Released schema lock
0625 08:04:28.313153 (+ 146us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.208251 (+895098us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.312951 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER)
0625 08:04:28.312960 (+ 9us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.312966 (+ 6us) write_op.cc:454] Released schema lock
0625 08:04:28.312969 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":894274,"Hive Metastore.run_cpu_time_us":476,"Hive Metastore.run_wall_time_us":128628,"spinlock_wait_cycles":25344,"child_traces":[["op",{}]]}
I20250625 08:04:29.211108 2118 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_a3a8c500599e482da5b7120ec26a797a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.215340 2222 tablet_bootstrap.cc:492] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.216202 2221 consensus_queue.cc:260] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.217353 2221 ts_tablet_manager.cc:1428] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.013s user 0.003s sys 0.002s
I20250625 08:04:29.217741 1894 heartbeater.cc:499] Master 127.1.19.124:37745 was elected leader, sending a full tablet report...
I20250625 08:04:29.217823 2222 tablet_bootstrap.cc:654] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.214681 2223 tablet_bootstrap.cc:654] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.221109 2223 log.cc:826] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:29.221853 2116 master_service.cc:948] ReplaceTablet: received request to replace tablet cd42120cc1ee43c1bd11944529eee1a5 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.224756 2222 log.cc:826] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:29.226527 2223 tablet_bootstrap.cc:492] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.226856 2223 ts_tablet_manager.cc:1397] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.002s
I20250625 08:04:29.228113 2115 master_service.cc:948] ReplaceTablet: received request to replace tablet cd42120cc1ee43c1bd11944529eee1a5 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.228312 2114 master_service.cc:948] ReplaceTablet: received request to replace tablet cd42120cc1ee43c1bd11944529eee1a5 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.228428 2113 master_service.cc:948] ReplaceTablet: received request to replace tablet cd42120cc1ee43c1bd11944529eee1a5 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.230906 2223 raft_consensus.cc:357] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.231118 2223 raft_consensus.cc:383] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.231153 2223 raft_consensus.cc:738] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.231570 2223 consensus_queue.cc:260] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.232496 2223 ts_tablet_manager.cc:1428] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250625 08:04:29.235338 2222 tablet_bootstrap.cc:492] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.235473 2116 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet cd42120cc1ee43c1bd11944529eee1a5
I20250625 08:04:29.235589 2222 ts_tablet_manager.cc:1397] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20250625 08:04:29.235857 2116 catalog_manager.cc:6652] ReplaceTablet: tablet cd42120cc1ee43c1bd11944529eee1a5 deleted and replaced by tablet 0e0fba9377e54618b87b2a018412d0a6
I20250625 08:04:29.238394 1828 tablet_service.cc:1515] Processing DeleteTablet for tablet cd42120cc1ee43c1bd11944529eee1a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e0fba9377e54618b87b2a018412d0a6) from {username='slave'} at 127.0.0.1:53818
I20250625 08:04:29.238610 1691 tablet_service.cc:1515] Processing DeleteTablet for tablet cd42120cc1ee43c1bd11944529eee1a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e0fba9377e54618b87b2a018412d0a6) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:29.239285 1963 tablet_service.cc:1515] Processing DeleteTablet for tablet cd42120cc1ee43c1bd11944529eee1a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e0fba9377e54618b87b2a018412d0a6) from {username='slave'} at 127.0.0.1:44472
I20250625 08:04:29.239734 2232 tablet_replica.cc:331] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:29.239728 2222 raft_consensus.cc:357] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.239982 2222 raft_consensus.cc:383] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 08:04:29.240046 2106 catalog_manager.cc:4908] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): delete failed for tablet cd42120cc1ee43c1bd11944529eee1a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd42120cc1ee43c1bd11944529eee1a5 already in progress: creating tablet
I20250625 08:04:29.240078 2232 raft_consensus.cc:2241] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.240046 2222 raft_consensus.cc:738] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.240365 2232 raft_consensus.cc:2270] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.240721 2222 consensus_queue.cc:260] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.241552 2232 ts_tablet_manager.cc:1905] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.241750 2222 ts_tablet_manager.cc:1428] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250625 08:04:29.244297 2232 ts_tablet_manager.cc:1918] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.244428 2232 log.cc:1199] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/cd42120cc1ee43c1bd11944529eee1a5
I20250625 08:04:29.244783 2232 ts_tablet_manager.cc:1939] T cd42120cc1ee43c1bd11944529eee1a5 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:29.244778 1691 tablet_service.cc:1515] Processing DeleteTablet for tablet cd42120cc1ee43c1bd11944529eee1a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e0fba9377e54618b87b2a018412d0a6) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:29.244972 2231 tablet_replica.cc:331] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:29.245262 2231 raft_consensus.cc:2241] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.245502 2231 raft_consensus.cc:2270] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.246271 1963 tablet_service.cc:1468] Processing CreateTablet for tablet 0e0fba9377e54618b87b2a018412d0a6 (DEFAULT_TABLE table=default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.246337 1828 tablet_service.cc:1468] Processing CreateTablet for tablet 0e0fba9377e54618b87b2a018412d0a6 (DEFAULT_TABLE table=default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.246574 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e0fba9377e54618b87b2a018412d0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.246634 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e0fba9377e54618b87b2a018412d0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.246646 2231 ts_tablet_manager.cc:1905] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.248490 2117 master_service.cc:948] ReplaceTablet: received request to replace tablet cd42120cc1ee43c1bd11944529eee1a5 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.248874 2231 ts_tablet_manager.cc:1918] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.248994 2231 log.cc:1199] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/cd42120cc1ee43c1bd11944529eee1a5
I20250625 08:04:29.249337 2231 ts_tablet_manager.cc:1939] T cd42120cc1ee43c1bd11944529eee1a5 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:29.249367 1691 tablet_service.cc:1468] Processing CreateTablet for tablet 0e0fba9377e54618b87b2a018412d0a6 (DEFAULT_TABLE table=default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.249445 2221 tablet_bootstrap.cc:492] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.249640 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e0fba9377e54618b87b2a018412d0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.250967 2221 tablet_bootstrap.cc:654] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.252743 2221 tablet_bootstrap.cc:492] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.252840 2221 ts_tablet_manager.cc:1397] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250625 08:04:29.252918 2222 tablet_bootstrap.cc:492] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.253501 2229 tablet_replica.cc:331] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:29.253469 2221 raft_consensus.cc:357] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.253585 2221 raft_consensus.cc:383] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.253628 2221 raft_consensus.cc:738] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.253772 2221 consensus_queue.cc:260] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.253968 2222 tablet_bootstrap.cc:654] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.254024 2221 ts_tablet_manager.cc:1428] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.255029 2229 raft_consensus.cc:2241] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.255075 2222 tablet_bootstrap.cc:492] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.255144 2222 ts_tablet_manager.cc:1397] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:29.255327 2229 raft_consensus.cc:2270] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.255595 2222 raft_consensus.cc:357] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.255676 2222 raft_consensus.cc:383] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.255704 2222 raft_consensus.cc:738] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.255843 2222 consensus_queue.cc:260] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.256086 2222 ts_tablet_manager.cc:1428] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.256709 1757 heartbeater.cc:499] Master 127.1.19.124:37745 was elected leader, sending a full tablet report...
I20250625 08:04:29.259712 2223 tablet_bootstrap.cc:492] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.260988 2223 tablet_bootstrap.cc:654] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.262257 2223 tablet_bootstrap.cc:492] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.262357 2223 ts_tablet_manager.cc:1397] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20250625 08:04:29.262954 2223 raft_consensus.cc:357] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.263067 2223 raft_consensus.cc:383] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.263095 2223 raft_consensus.cc:738] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.263185 2106 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet cd42120cc1ee43c1bd11944529eee1a5 (table default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:29.263229 2223 consensus_queue.cc:260] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.263490 2223 ts_tablet_manager.cc:1428] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 08:04:29.264170 2229 ts_tablet_manager.cc:1905] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.264185 2029 heartbeater.cc:499] Master 127.1.19.124:37745 was elected leader, sending a full tablet report...
I20250625 08:04:29.266673 2229 ts_tablet_manager.cc:1918] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.266851 2229 log.cc:1199] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/cd42120cc1ee43c1bd11944529eee1a5
I20250625 08:04:29.266960 2119 catalog_manager.cc:2149] created HMS catalog entry for table [id=1a45dede976343d1be49292f6353b875]
I20250625 08:04:29.276284 2182 catalog_manager.cc:2747] Deleting table default.table_a3a8c500599e482da5b7120ec26a797a [id=24b00899d74d48f5866b7d034091f802] in response to Hive Metastore notification log event 14
W20250625 08:04:29.276463 2182 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 14 DROP_TABLE default.table_a3a8c500599e482da5b7120ec26a797a: Not found: the table does not exist: table_id: "24b00899d74d48f5866b7d034091f802" table_name: "default.table_a3a8c500599e482da5b7120ec26a797a"
I20250625 08:04:29.277642 2106 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet cd42120cc1ee43c1bd11944529eee1a5 (table default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:29.282075 1691 tablet_service.cc:1468] Processing CreateTablet for tablet fca9393618ef47bbbd0d4ffb34e862c1 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.282423 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca9393618ef47bbbd0d4ffb34e862c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.282730 1963 tablet_service.cc:1468] Processing CreateTablet for tablet fca9393618ef47bbbd0d4ffb34e862c1 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.282999 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca9393618ef47bbbd0d4ffb34e862c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.282979 2119 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_b49cbf7ee55042b39d7b5d3946a35618" } new_table_name: "default.Table_7cb71508c2934e1aa091c71056ae7a01" modify_external_catalogs: true
W20250625 08:04:29.283023 2118 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a3a8c500599e482da5b7120ec26a797a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:04:29.284760 2222 tablet_bootstrap.cc:492] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.285506 2223 tablet_bootstrap.cc:492] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.285949 2222 tablet_bootstrap.cc:654] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.286492 2223 tablet_bootstrap.cc:654] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.287300 2222 tablet_bootstrap.cc:492] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.287387 2222 ts_tablet_manager.cc:1397] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:29.288002 2222 raft_consensus.cc:357] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.288126 2222 raft_consensus.cc:383] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.288159 2222 raft_consensus.cc:738] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.288281 2222 consensus_queue.cc:260] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.288534 2222 ts_tablet_manager.cc:1428] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.273466 2225 raft_consensus.cc:491] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 08:04:29.274108 1897 tablet.cc:2378] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 08:04:29.291079 2225 raft_consensus.cc:513] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.291257 2229 ts_tablet_manager.cc:1939] T cd42120cc1ee43c1bd11944529eee1a5 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:29.282274 1828 tablet_service.cc:1468] Processing CreateTablet for tablet fca9393618ef47bbbd0d4ffb34e862c1 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.291605 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca9393618ef47bbbd0d4ffb34e862c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.292346 2225 leader_election.cc:290] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:29.293910 2106 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet cd42120cc1ee43c1bd11944529eee1a5 (table default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:29.294188 2221 tablet_bootstrap.cc:492] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.295221 2221 tablet_bootstrap.cc:654] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.296213 2223 tablet_bootstrap.cc:492] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.296298 2223 ts_tablet_manager.cc:1397] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20250625 08:04:29.296774 2223 raft_consensus.cc:357] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.296885 2223 raft_consensus.cc:383] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.296916 2223 raft_consensus.cc:738] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.297032 2223 consensus_queue.cc:260] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.297281 2223 ts_tablet_manager.cc:1428] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.297302 2221 tablet_bootstrap.cc:492] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.297367 2221 ts_tablet_manager.cc:1397] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:29.297724 2221 raft_consensus.cc:357] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.297808 2221 raft_consensus.cc:383] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.297865 2221 raft_consensus.cc:738] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.297998 2221 consensus_queue.cc:260] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.298238 2221 ts_tablet_manager.cc:1428] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.304828 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e0fba9377e54618b87b2a018412d0a6" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:29.305164 1711 raft_consensus.cc:2466] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:29.305662 1784 leader_election.cc:304] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:29.305949 2225 raft_consensus.cc:2802] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:29.306044 2225 raft_consensus.cc:491] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:29.306105 2225 raft_consensus.cc:3058] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.307286 2225 raft_consensus.cc:513] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.307696 2225 leader_election.cc:290] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:29.308079 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e0fba9377e54618b87b2a018412d0a6" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:29.308213 1711 raft_consensus.cc:3058] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.309165 1711 raft_consensus.cc:2466] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:29.310046 1784 leader_election.cc:304] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:29.310746 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e0fba9377e54618b87b2a018412d0a6" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:29.310873 2225 raft_consensus.cc:2802] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:29.311053 1983 raft_consensus.cc:2466] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:29.311110 2225 raft_consensus.cc:695] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:29.310741 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e0fba9377e54618b87b2a018412d0a6" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:29.311409 1982 raft_consensus.cc:3058] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.311391 2225 consensus_queue.cc:237] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.312351 1982 raft_consensus.cc:2466] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:29.314517 2111 catalog_manager.cc:5582] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf reported cstate change: term changed from 0 to 1, leader changed from <none> to de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66). New cstate: current_term: 1 leader_uuid: "de62034f80d9463dbd6386fc3e90fdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } health_report { overall_health: HEALTHY } } }
W20250625 08:04:29.323181 1760 tablet.cc:2378] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 08:04:29.363602 2111 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_88079d64160a4318bf8ea1807cb3a30d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.377815 1442 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=93ac3d33131249dd809793b74ef4523f]
I20250625 08:04:29.378144 1442 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 31) took 1751 ms. Trace:
I20250625 08:04:29.378206 1442 rpcz_store.cc:276] 0625 08:04:27.626635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626833 (+ 198us) service_pool.cc:225] Handling call
0625 08:04:27.654949 (+ 28116us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:27.656113 (+ 1164us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.173048 (+516935us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.330206 (+157158us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.378111 (+1047905us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.174203 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 08:04:28.174315 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0625 08:04:28.174320 (+ 5us) write_op.cc:435] Acquired schema lock
0625 08:04:28.174321 (+ 1us) tablet.cc:660] Decoding operations
0625 08:04:28.174411 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0625 08:04:28.174422 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0625 08:04:28.174425 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0625 08:04:28.174647 (+ 222us) tablet.cc:699] Row locks acquired
0625 08:04:28.174649 (+ 2us) write_op.cc:260] PREPARE: finished
0625 08:04:28.174693 (+ 44us) write_op.cc:270] Start()
0625 08:04:28.174722 (+ 29us) write_op.cc:276] Timestamp: P: 1750838668174671 usec, L: 0
0625 08:04:28.174726 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 08:04:28.174894 (+ 168us) log.cc:844] Serialized 432 byte log entry
0625 08:04:28.329417 (+154523us) op_driver.cc:464] REPLICATION: finished
0625 08:04:28.329430 (+ 13us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0625 08:04:28.329434 (+ 4us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.329466 (+ 32us) write_op.cc:454] Released schema lock
0625 08:04:28.329473 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":873896,"Hive Metastore.run_cpu_time_us":626,"Hive Metastore.run_wall_time_us":689958,"spinlock_wait_cycles":17792,"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":698,"prepare.run_wall_time_us":792,"raft.queue_time_us":820,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":203,"replication_time_us":154627}]]}
I20250625 08:04:29.380818 2112 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_d28f89eb30604a7abf867f68131b2c1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.434940 1440 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=91897f72c011456fb9b3fa2abb08b6b1]
I20250625 08:04:29.435249 1440 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 33) took 1808 ms. Trace:
I20250625 08:04:29.435310 1440 rpcz_store.cc:276] 0625 08:04:27.626734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626794 (+ 60us) service_pool.cc:225] Handling call
0625 08:04:28.179653 (+552859us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:28.179963 (+ 310us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.354696 (+174733us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.355337 (+ 641us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.355342 (+ 5us) write_op.cc:454] Released schema lock
0625 08:04:28.355463 (+ 121us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.435225 (+1079762us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.355264 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER)
0625 08:04:28.355271 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.355274 (+ 3us) write_op.cc:454] Released schema lock
0625 08:04:28.355276 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1154592,"Hive Metastore.run_cpu_time_us":461,"Hive Metastore.run_wall_time_us":99235,"mutex_wait_us":342,"child_traces":[["op",{}]]}
I20250625 08:04:29.438012 2113 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.450791 2115 master_service.cc:948] ReplaceTablet: received request to replace tablet fca9393618ef47bbbd0d4ffb34e862c1 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.457496 2114 master_service.cc:948] ReplaceTablet: received request to replace tablet 0e0fba9377e54618b87b2a018412d0a6 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.458381 2117 master_service.cc:948] ReplaceTablet: received request to replace tablet 0e0fba9377e54618b87b2a018412d0a6 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.458583 2116 master_service.cc:948] ReplaceTablet: received request to replace tablet fca9393618ef47bbbd0d4ffb34e862c1 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.460179 2115 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet fca9393618ef47bbbd0d4ffb34e862c1
I20250625 08:04:29.460481 2115 catalog_manager.cc:6652] ReplaceTablet: tablet fca9393618ef47bbbd0d4ffb34e862c1 deleted and replaced by tablet 9deb9ceefd014d0996dd65fefb10e659
I20250625 08:04:29.460708 2115 master_service.cc:948] ReplaceTablet: received request to replace tablet fca9393618ef47bbbd0d4ffb34e862c1 from {username='slave'} at 127.0.0.1:57934
W20250625 08:04:29.462636 2032 tablet.cc:2378] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 08:04:29.462736 1963 tablet_service.cc:1515] Processing DeleteTablet for tablet fca9393618ef47bbbd0d4ffb34e862c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9deb9ceefd014d0996dd65fefb10e659) from {username='slave'} at 127.0.0.1:44472
I20250625 08:04:29.462940 2232 tablet_replica.cc:331] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:29.463358 2232 raft_consensus.cc:2241] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.463487 2232 raft_consensus.cc:2270] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.464650 1828 tablet_service.cc:1515] Processing DeleteTablet for tablet fca9393618ef47bbbd0d4ffb34e862c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9deb9ceefd014d0996dd65fefb10e659) from {username='slave'} at 127.0.0.1:53818
I20250625 08:04:29.464843 2229 tablet_replica.cc:331] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:29.464962 2229 raft_consensus.cc:2241] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.465075 2229 raft_consensus.cc:2270] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.465276 1691 tablet_service.cc:1515] Processing DeleteTablet for tablet fca9393618ef47bbbd0d4ffb34e862c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9deb9ceefd014d0996dd65fefb10e659) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:29.465528 2231 tablet_replica.cc:331] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:29.465771 2231 raft_consensus.cc:2241] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.465901 2231 raft_consensus.cc:2270] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.466665 2229 ts_tablet_manager.cc:1905] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.469223 2229 ts_tablet_manager.cc:1918] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.469311 2229 log.cc:1199] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/fca9393618ef47bbbd0d4ffb34e862c1
I20250625 08:04:29.469537 2229 ts_tablet_manager.cc:1939] T fca9393618ef47bbbd0d4ffb34e862c1 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:29.470233 2232 ts_tablet_manager.cc:1905] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.472620 2232 ts_tablet_manager.cc:1918] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.472705 2232 log.cc:1199] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/fca9393618ef47bbbd0d4ffb34e862c1
I20250625 08:04:29.472945 2232 ts_tablet_manager.cc:1939] T fca9393618ef47bbbd0d4ffb34e862c1 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:29.474220 2231 ts_tablet_manager.cc:1905] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.476162 2231 ts_tablet_manager.cc:1918] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.476239 2231 log.cc:1199] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/fca9393618ef47bbbd0d4ffb34e862c1
I20250625 08:04:29.476418 2231 ts_tablet_manager.cc:1939] T fca9393618ef47bbbd0d4ffb34e862c1 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:29.476509 2110 catalog_manager.cc:3460] renamed table default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089] in HMS: new name default.table_23c54834c68a47bd9654b34281b53e7d
I20250625 08:04:29.477594 2106 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet fca9393618ef47bbbd0d4ffb34e862c1 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:29.477788 2106 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet fca9393618ef47bbbd0d4ffb34e862c1 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:29.480314 2182 catalog_manager.cc:2747] Deleting table default.table_d28f89eb30604a7abf867f68131b2c1a [id=93ac3d33131249dd809793b74ef4523f] in response to Hive Metastore notification log event 16
W20250625 08:04:29.480495 2182 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 16 DROP_TABLE default.table_d28f89eb30604a7abf867f68131b2c1a: Not found: the table does not exist: table_id: "93ac3d33131249dd809793b74ef4523f" table_name: "default.table_d28f89eb30604a7abf867f68131b2c1a"
I20250625 08:04:29.480834 2182 catalog_manager.cc:2747] Deleting table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=91897f72c011456fb9b3fa2abb08b6b1] in response to Hive Metastore notification log event 17
W20250625 08:04:29.480967 2182 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_c06b3617aaf24ba4b0ae0b3926081026: Not found: the table does not exist: table_id: "91897f72c011456fb9b3fa2abb08b6b1" table_name: "default.table_c06b3617aaf24ba4b0ae0b3926081026"
I20250625 08:04:29.489228 2116 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_5eebaa7c05694491963fce48525e4681"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.495733 2114 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet 0e0fba9377e54618b87b2a018412d0a6
I20250625 08:04:29.496076 2114 catalog_manager.cc:6652] ReplaceTablet: tablet 0e0fba9377e54618b87b2a018412d0a6 deleted and replaced by tablet d25d775165474789bd7f87d39bdf5e5f
I20250625 08:04:29.498615 1828 tablet_service.cc:1468] Processing CreateTablet for tablet 9deb9ceefd014d0996dd65fefb10e659 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.498746 1827 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e0fba9377e54618b87b2a018412d0a6 with delete_type TABLET_DATA_DELETED (replaced by tablet d25d775165474789bd7f87d39bdf5e5f) from {username='slave'} at 127.0.0.1:53818
I20250625 08:04:29.498953 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9deb9ceefd014d0996dd65fefb10e659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.501333 2221 tablet_bootstrap.cc:492] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.502674 1447 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=76655d21216448fb8307987100f8513f]
I20250625 08:04:29.502921 1447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 27) took 1876 ms. Trace:
I20250625 08:04:29.502974 1447 rpcz_store.cc:276] 0625 08:04:27.626255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626317 (+ 62us) service_pool.cc:225] Handling call
0625 08:04:28.180104 (+553787us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:28.180377 (+ 273us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.402428 (+222051us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.403066 (+ 638us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.403068 (+ 2us) write_op.cc:454] Released schema lock
0625 08:04:28.403187 (+ 119us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.502903 (+1099716us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.403018 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER)
0625 08:04:28.403027 (+ 9us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.403032 (+ 5us) write_op.cc:454] Released schema lock
0625 08:04:28.403035 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1205804,"Hive Metastore.run_cpu_time_us":426,"Hive Metastore.run_wall_time_us":115459,"mutex_wait_us":181,"spinlock_wait_cycles":754560,"child_traces":[["op",{}]]}
I20250625 08:04:29.503088 1690 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e0fba9377e54618b87b2a018412d0a6 with delete_type TABLET_DATA_DELETED (replaced by tablet d25d775165474789bd7f87d39bdf5e5f) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:29.503443 1691 tablet_service.cc:1468] Processing CreateTablet for tablet 9deb9ceefd014d0996dd65fefb10e659 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.503718 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9deb9ceefd014d0996dd65fefb10e659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.506125 2222 tablet_bootstrap.cc:492] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.506501 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e0fba9377e54618b87b2a018412d0a6 with delete_type TABLET_DATA_DELETED (replaced by tablet d25d775165474789bd7f87d39bdf5e5f) from {username='slave'} at 127.0.0.1:44472
I20250625 08:04:29.506850 1963 tablet_service.cc:1468] Processing CreateTablet for tablet 9deb9ceefd014d0996dd65fefb10e659 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.507134 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9deb9ceefd014d0996dd65fefb10e659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.507302 2222 tablet_bootstrap.cc:654] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.509435 2223 tablet_bootstrap.cc:492] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.510543 2223 tablet_bootstrap.cc:654] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.512014 2223 tablet_bootstrap.cc:492] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.512122 2223 ts_tablet_manager.cc:1397] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250625 08:04:29.512632 2223 raft_consensus.cc:357] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.512742 2223 raft_consensus.cc:383] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.512777 2223 raft_consensus.cc:738] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.512740 2117 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.512908 2223 consensus_queue.cc:260] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.513202 2223 ts_tablet_manager.cc:1428] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:29.516984 2221 tablet_bootstrap.cc:654] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.517340 2118 catalog_manager.cc:2149] created HMS catalog entry for table [id=06139221a5c54ced9f9903b3c0c9f020]
I20250625 08:04:29.518590 2221 tablet_bootstrap.cc:492] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.518708 2221 ts_tablet_manager.cc:1397] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250625 08:04:29.519255 2221 raft_consensus.cc:357] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.519382 2221 raft_consensus.cc:383] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.519418 2221 raft_consensus.cc:738] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.519555 2221 consensus_queue.cc:260] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.520298 2221 ts_tablet_manager.cc:1428] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 08:04:29.533896 2118 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_a3a8c500599e482da5b7120ec26a797a" } new_table_name: "default.Table_6894ca710ded4040ba337309d454d090" modify_external_catalogs: true
I20250625 08:04:29.535403 1691 tablet_service.cc:1468] Processing CreateTablet for tablet d25d775165474789bd7f87d39bdf5e5f (DEFAULT_TABLE table=default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.535903 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25d775165474789bd7f87d39bdf5e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.535980 1963 tablet_service.cc:1468] Processing CreateTablet for tablet d25d775165474789bd7f87d39bdf5e5f (DEFAULT_TABLE table=default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.536293 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25d775165474789bd7f87d39bdf5e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.538913 1828 tablet_service.cc:1468] Processing CreateTablet for tablet d25d775165474789bd7f87d39bdf5e5f (DEFAULT_TABLE table=default.table_c1ad5f1d55214059b710e6acaa0f61d6 [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.540611 2231 tablet_replica.cc:331] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:29.540812 2231 raft_consensus.cc:2241] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.540937 2231 raft_consensus.cc:2270] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.541476 2231 ts_tablet_manager.cc:1905] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.543843 2231 ts_tablet_manager.cc:1918] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.544010 2231 log.cc:1199] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/0e0fba9377e54618b87b2a018412d0a6
I20250625 08:04:29.544294 2231 ts_tablet_manager.cc:1939] T 0e0fba9377e54618b87b2a018412d0a6 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
W20250625 08:04:29.544358 2182 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d25d775165474789bd7f87d39bdf5e5f consensus state has no leader
I20250625 08:04:29.547725 2106 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 0e0fba9377e54618b87b2a018412d0a6 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:29.550704 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25d775165474789bd7f87d39bdf5e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.552263 2106 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet fca9393618ef47bbbd0d4ffb34e862c1 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:29.557466 2182 catalog_manager.cc:2747] Deleting table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=76655d21216448fb8307987100f8513f] in response to Hive Metastore notification log event 19
W20250625 08:04:29.557669 2182 hms_notification_log_listener.cc:329] Failed to handle Hive Metastore notification: 19 DROP_TABLE default.table_b0a79766b44a4a6882c263b6c2c46ed2: Not found: the table does not exist: table_id: "76655d21216448fb8307987100f8513f" table_name: "default.table_b0a79766b44a4a6882c263b6c2c46ed2"
I20250625 08:04:29.560214 1691 tablet_service.cc:1468] Processing CreateTablet for tablet 35c5be3d39c34774bcabfd66aaf8dbaa (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.560214 1827 tablet_service.cc:1468] Processing CreateTablet for tablet 35c5be3d39c34774bcabfd66aaf8dbaa (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.560544 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c5be3d39c34774bcabfd66aaf8dbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.560542 1827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c5be3d39c34774bcabfd66aaf8dbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.563038 1962 tablet_service.cc:1468] Processing CreateTablet for tablet 35c5be3d39c34774bcabfd66aaf8dbaa (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.563369 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c5be3d39c34774bcabfd66aaf8dbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.565609 2223 tablet_bootstrap.cc:492] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.566223 2221 tablet_bootstrap.cc:492] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.566759 2223 tablet_bootstrap.cc:654] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.567200 2221 tablet_bootstrap.cc:654] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.568224 2232 tablet_replica.cc:331] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:29.568351 2232 raft_consensus.cc:2241] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.568473 2232 raft_consensus.cc:2270] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.568673 2229 tablet_replica.cc:331] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:29.568783 2229 raft_consensus.cc:2241] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:29.569016 2232 ts_tablet_manager.cc:1905] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.569036 2229 pending_rounds.cc:70] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Trying to abort 1 pending ops.
I20250625 08:04:29.569162 2229 pending_rounds.cc:77] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171435189501734912 op_type: NO_OP noop_request { }
I20250625 08:04:29.569275 2229 raft_consensus.cc:2887] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 08:04:29.569350 2229 raft_consensus.cc:2270] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.570564 2229 ts_tablet_manager.cc:1905] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.575054 2229 ts_tablet_manager.cc:1918] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:29.576655 2232 ts_tablet_manager.cc:1918] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.576823 2232 log.cc:1199] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/0e0fba9377e54618b87b2a018412d0a6
W20250625 08:04:29.576797 2111 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_88079d64160a4318bf8ea1807cb3a30d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:04:29.577122 2232 ts_tablet_manager.cc:1939] T 0e0fba9377e54618b87b2a018412d0a6 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
W20250625 08:04:29.577231 2104 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet d25d775165474789bd7f87d39bdf5e5f (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d25d775165474789bd7f87d39bdf5e5f consensus state has no leader
I20250625 08:04:29.577382 2229 log.cc:1199] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/0e0fba9377e54618b87b2a018412d0a6
W20250625 08:04:29.577582 2112 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d28f89eb30604a7abf867f68131b2c1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:04:29.577956 2229 ts_tablet_manager.cc:1939] T 0e0fba9377e54618b87b2a018412d0a6 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
W20250625 08:04:29.578315 2113 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c06b3617aaf24ba4b0ae0b3926081026 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:04:29.579193 2106 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 0e0fba9377e54618b87b2a018412d0a6 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:29.579526 2222 tablet_bootstrap.cc:492] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.579527 2106 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet 0e0fba9377e54618b87b2a018412d0a6 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:29.579635 2222 ts_tablet_manager.cc:1397] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.000s
W20250625 08:04:29.579639 2116 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5eebaa7c05694491963fce48525e4681 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 08:04:29.580282 2117 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b0a79766b44a4a6882c263b6c2c46ed2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:04:29.580228 2222 raft_consensus.cc:357] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.580348 2222 raft_consensus.cc:383] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.580405 2222 raft_consensus.cc:738] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.580564 2222 consensus_queue.cc:260] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.580852 2222 ts_tablet_manager.cc:1428] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.581076 2222 tablet_bootstrap.cc:492] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.570653 2221 tablet_bootstrap.cc:492] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.582051 2221 ts_tablet_manager.cc:1397] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250625 08:04:29.582129 2222 tablet_bootstrap.cc:654] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.582609 2221 raft_consensus.cc:357] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.582737 2221 raft_consensus.cc:383] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.582782 2221 raft_consensus.cc:738] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.582965 2221 consensus_queue.cc:260] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.583335 2221 ts_tablet_manager.cc:1428] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.583499 2221 tablet_bootstrap.cc:492] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.584564 2221 tablet_bootstrap.cc:654] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.585135 2223 tablet_bootstrap.cc:492] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.585244 2223 ts_tablet_manager.cc:1397] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250625 08:04:29.585698 2223 raft_consensus.cc:357] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.585803 2223 raft_consensus.cc:383] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.585834 2223 raft_consensus.cc:738] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.585961 2223 consensus_queue.cc:260] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.586234 2223 ts_tablet_manager.cc:1428] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.586457 2223 tablet_bootstrap.cc:492] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.587643 2223 tablet_bootstrap.cc:654] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.591279 2110 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_23c54834c68a47bd9654b34281b53e7d" } new_table_name: "default.Table_fbd9fab6fc2a480f88e12bb1ada0b9f6" modify_external_catalogs: true
I20250625 08:04:29.592142 2223 tablet_bootstrap.cc:492] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.592248 2223 ts_tablet_manager.cc:1397] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 08:04:29.592700 2223 raft_consensus.cc:357] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.592804 2223 raft_consensus.cc:383] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.592840 2223 raft_consensus.cc:738] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.592975 2223 consensus_queue.cc:260] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.594098 2222 tablet_bootstrap.cc:492] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.594209 2222 ts_tablet_manager.cc:1397] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20250625 08:04:29.595453 2114 catalog_manager.cc:5674] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005) does not have the latest schema for tablet d25d775165474789bd7f87d39bdf5e5f (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
I20250625 08:04:29.595497 2223 ts_tablet_manager.cc:1428] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 08:04:29.597105 2114 catalog_manager.cc:5674] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005) does not have the latest schema for tablet d25d775165474789bd7f87d39bdf5e5f (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
I20250625 08:04:29.594893 2222 raft_consensus.cc:357] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.597795 2222 raft_consensus.cc:383] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.597882 2222 raft_consensus.cc:738] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.598069 2222 consensus_queue.cc:260] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.599220 2221 tablet_bootstrap.cc:492] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.599334 2221 ts_tablet_manager.cc:1397] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250625 08:04:29.599848 2221 raft_consensus.cc:357] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.599969 2221 raft_consensus.cc:383] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.600009 2221 raft_consensus.cc:738] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.600147 2221 consensus_queue.cc:260] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.600435 2221 ts_tablet_manager.cc:1428] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.601483 2114 catalog_manager.cc:5674] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085) does not have the latest schema for tablet d25d775165474789bd7f87d39bdf5e5f (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
I20250625 08:04:29.602039 2222 ts_tablet_manager.cc:1428] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250625 08:04:29.604604 2222 tablet_bootstrap.cc:492] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.604985 2115 catalog_manager.cc:5674] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085) does not have the latest schema for tablet d25d775165474789bd7f87d39bdf5e5f (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
I20250625 08:04:29.606007 2228 raft_consensus.cc:491] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:29.606141 2228 raft_consensus.cc:513] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.606523 2114 catalog_manager.cc:5674] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071) does not have the latest schema for tablet d25d775165474789bd7f87d39bdf5e5f (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
I20250625 08:04:29.607549 2228 leader_election.cc:290] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.607682 1444 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=7825ea67af2d42b68200097030b08116]
I20250625 08:04:29.607935 1444 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 30) took 1981 ms. Trace:
I20250625 08:04:29.607981 1444 rpcz_store.cc:276] 0625 08:04:27.626409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626488 (+ 79us) service_pool.cc:225] Handling call
0625 08:04:28.180298 (+553810us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:28.180582 (+ 284us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.434422 (+253840us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.435071 (+ 649us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.435073 (+ 2us) write_op.cc:454] Released schema lock
0625 08:04:28.435190 (+ 117us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.607922 (+1172732us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.435018 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER)
0625 08:04:28.435028 (+ 10us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.435033 (+ 5us) write_op.cc:454] Released schema lock
0625 08:04:28.435037 (+ 4us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1289159,"Hive Metastore.run_cpu_time_us":463,"Hive Metastore.run_wall_time_us":136808,"child_traces":[["op",{}]]}
I20250625 08:04:29.608440 2222 tablet_bootstrap.cc:654] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.610285 2115 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_31023e55a4384ac88f8415871e52d142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.614135 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9deb9ceefd014d0996dd65fefb10e659" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:29.614300 1711 raft_consensus.cc:2466] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:29.614972 1919 leader_election.cc:304] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:29.615286 2228 raft_consensus.cc:2802] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:29.615370 2228 raft_consensus.cc:491] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:29.615413 2228 raft_consensus.cc:3058] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.616601 2228 raft_consensus.cc:513] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.617038 2228 leader_election.cc:290] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.618117 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9deb9ceefd014d0996dd65fefb10e659" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:29.618260 1711 raft_consensus.cc:3058] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.619417 1711 raft_consensus.cc:2466] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:29.619985 1919 leader_election.cc:304] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:29.620267 2228 raft_consensus.cc:2802] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:29.620494 2228 raft_consensus.cc:695] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:29.620796 2228 consensus_queue.cc:237] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.624526 2114 catalog_manager.cc:5582] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67). New cstate: current_term: 1 leader_uuid: "5dae8595b9484b57bc7ccea543f83130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } health_report { overall_health: UNKNOWN } } }
I20250625 08:04:29.624668 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9deb9ceefd014d0996dd65fefb10e659" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:29.624893 1848 raft_consensus.cc:2466] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:29.625171 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9deb9ceefd014d0996dd65fefb10e659" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:29.625275 1847 raft_consensus.cc:3058] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.626173 1847 raft_consensus.cc:2466] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:29.632438 2222 tablet_bootstrap.cc:492] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.632551 2222 ts_tablet_manager.cc:1397] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.002s
I20250625 08:04:29.633009 2222 raft_consensus.cc:357] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.633111 2222 raft_consensus.cc:383] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.633165 2222 raft_consensus.cc:738] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.633313 2222 consensus_queue.cc:260] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.633550 2222 ts_tablet_manager.cc:1428] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.651189 2256 raft_consensus.cc:491] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:29.651356 2256 raft_consensus.cc:513] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.651852 2256 leader_election.cc:290] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.652688 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35c5be3d39c34774bcabfd66aaf8dbaa" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:29.652688 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35c5be3d39c34774bcabfd66aaf8dbaa" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:29.652832 1847 raft_consensus.cc:2466] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:29.652832 1711 raft_consensus.cc:2466] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:29.653252 1919 leader_election.cc:304] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:29.653713 2256 raft_consensus.cc:2802] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:29.653774 2256 raft_consensus.cc:491] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:29.653810 2256 raft_consensus.cc:3058] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.654954 2256 raft_consensus.cc:513] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.655344 2256 leader_election.cc:290] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.656083 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35c5be3d39c34774bcabfd66aaf8dbaa" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:29.656234 1847 raft_consensus.cc:3058] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.656222 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35c5be3d39c34774bcabfd66aaf8dbaa" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:29.656339 1711 raft_consensus.cc:3058] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.657240 1847 raft_consensus.cc:2466] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:29.657240 1711 raft_consensus.cc:2466] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:29.657728 1919 leader_election.cc:304] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:29.658103 2256 raft_consensus.cc:2802] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:29.658201 2256 raft_consensus.cc:695] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:29.658346 2256 consensus_queue.cc:237] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.661885 2114 catalog_manager.cc:5582] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67). New cstate: current_term: 1 leader_uuid: "5dae8595b9484b57bc7ccea543f83130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } health_report { overall_health: UNKNOWN } } }
I20250625 08:04:29.664744 2119 catalog_manager.cc:3460] renamed table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875] in HMS: new name default.table_7cb71508c2934e1aa091c71056ae7a01
I20250625 08:04:29.681069 2114 master_service.cc:948] ReplaceTablet: received request to replace tablet 35c5be3d39c34774bcabfd66aaf8dbaa from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.686215 2114 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet 35c5be3d39c34774bcabfd66aaf8dbaa
I20250625 08:04:29.686604 2114 catalog_manager.cc:6652] ReplaceTablet: tablet 35c5be3d39c34774bcabfd66aaf8dbaa deleted and replaced by tablet a24a0ae1aa5c40e4b1a87428190b1932
I20250625 08:04:29.686766 2114 master_service.cc:948] ReplaceTablet: received request to replace tablet d25d775165474789bd7f87d39bdf5e5f from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.688664 1691 tablet_service.cc:1515] Processing DeleteTablet for tablet 35c5be3d39c34774bcabfd66aaf8dbaa with delete_type TABLET_DATA_DELETED (replaced by tablet a24a0ae1aa5c40e4b1a87428190b1932) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:29.688897 1828 tablet_service.cc:1515] Processing DeleteTablet for tablet 35c5be3d39c34774bcabfd66aaf8dbaa with delete_type TABLET_DATA_DELETED (replaced by tablet a24a0ae1aa5c40e4b1a87428190b1932) from {username='slave'} at 127.0.0.1:53818
I20250625 08:04:29.689109 2231 tablet_replica.cc:331] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:29.689177 2229 tablet_replica.cc:331] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:29.689257 2231 raft_consensus.cc:2241] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.689383 2231 raft_consensus.cc:2270] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.689373 1963 tablet_service.cc:1515] Processing DeleteTablet for tablet 35c5be3d39c34774bcabfd66aaf8dbaa with delete_type TABLET_DATA_DELETED (replaced by tablet a24a0ae1aa5c40e4b1a87428190b1932) from {username='slave'} at 127.0.0.1:44472
I20250625 08:04:29.689577 2232 tablet_replica.cc:331] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:29.689743 2232 raft_consensus.cc:2241] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:29.689414 2229 raft_consensus.cc:2241] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.690088 2231 ts_tablet_manager.cc:1905] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.690197 2229 raft_consensus.cc:2270] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.690517 2232 pending_rounds.cc:70] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: Trying to abort 1 pending ops.
I20250625 08:04:29.691009 2229 ts_tablet_manager.cc:1905] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.692301 2231 ts_tablet_manager.cc:1918] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.692371 2231 log.cc:1199] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/35c5be3d39c34774bcabfd66aaf8dbaa
I20250625 08:04:29.692579 2231 ts_tablet_manager.cc:1939] T 35c5be3d39c34774bcabfd66aaf8dbaa P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:29.694283 2229 ts_tablet_manager.cc:1918] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.694427 2229 log.cc:1199] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/35c5be3d39c34774bcabfd66aaf8dbaa
I20250625 08:04:29.694693 2229 ts_tablet_manager.cc:1939] T 35c5be3d39c34774bcabfd66aaf8dbaa P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:29.695355 2106 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 35c5be3d39c34774bcabfd66aaf8dbaa (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]) successfully deleted
I20250625 08:04:29.693868 2232 pending_rounds.cc:77] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171435190925475840 op_type: NO_OP noop_request { }
I20250625 08:04:29.696632 2232 raft_consensus.cc:2887] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 08:04:29.697126 2106 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 35c5be3d39c34774bcabfd66aaf8dbaa (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]) successfully deleted
I20250625 08:04:29.697556 2232 raft_consensus.cc:2270] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.699499 2232 ts_tablet_manager.cc:1905] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.701719 2233 raft_consensus.cc:491] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:29.701846 2232 ts_tablet_manager.cc:1918] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:29.701844 2233 raft_consensus.cc:513] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.701920 2232 log.cc:1199] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/35c5be3d39c34774bcabfd66aaf8dbaa
I20250625 08:04:29.702170 2232 ts_tablet_manager.cc:1939] T 35c5be3d39c34774bcabfd66aaf8dbaa P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:29.696094 2114 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet d25d775165474789bd7f87d39bdf5e5f
I20250625 08:04:29.702952 2114 catalog_manager.cc:6652] ReplaceTablet: tablet d25d775165474789bd7f87d39bdf5e5f deleted and replaced by tablet 626d783b9280477a8dd9ee5cc7f79237
I20250625 08:04:29.703117 2114 master_service.cc:948] ReplaceTablet: received request to replace tablet 35c5be3d39c34774bcabfd66aaf8dbaa from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.703363 2233 leader_election.cc:290] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.703425 2114 master_service.cc:948] ReplaceTablet: received request to replace tablet 9deb9ceefd014d0996dd65fefb10e659 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.704095 2111 catalog_manager.cc:2149] created HMS catalog entry for table [id=7c7e3cbdaf064697a864873ed87a21b6]
I20250625 08:04:29.705056 1963 tablet_service.cc:1515] Processing DeleteTablet for tablet d25d775165474789bd7f87d39bdf5e5f with delete_type TABLET_DATA_DELETED (replaced by tablet 626d783b9280477a8dd9ee5cc7f79237) from {username='slave'} at 127.0.0.1:44472
I20250625 08:04:29.705235 1691 tablet_service.cc:1515] Processing DeleteTablet for tablet d25d775165474789bd7f87d39bdf5e5f with delete_type TABLET_DATA_DELETED (replaced by tablet 626d783b9280477a8dd9ee5cc7f79237) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:29.705405 1828 tablet_service.cc:1515] Processing DeleteTablet for tablet d25d775165474789bd7f87d39bdf5e5f with delete_type TABLET_DATA_DELETED (replaced by tablet 626d783b9280477a8dd9ee5cc7f79237) from {username='slave'} at 127.0.0.1:53818
I20250625 08:04:29.705775 2231 tablet_replica.cc:331] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:29.705839 2229 tablet_replica.cc:331] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:29.705938 2231 raft_consensus.cc:2241] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.705950 2229 raft_consensus.cc:2241] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.706053 2229 raft_consensus.cc:2270] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.706053 2231 raft_consensus.cc:2270] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.706634 2231 ts_tablet_manager.cc:1905] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.707136 2229 ts_tablet_manager.cc:1905] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.708870 2231 ts_tablet_manager.cc:1918] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.708935 2231 log.cc:1199] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/d25d775165474789bd7f87d39bdf5e5f
I20250625 08:04:29.709131 2231 ts_tablet_manager.cc:1939] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:29.709566 2229 ts_tablet_manager.cc:1918] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.709630 2229 log.cc:1199] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/d25d775165474789bd7f87d39bdf5e5f
I20250625 08:04:29.709805 2232 tablet_replica.cc:331] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:29.709820 2229 ts_tablet_manager.cc:1939] T d25d775165474789bd7f87d39bdf5e5f P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:29.709929 2232 raft_consensus.cc:2241] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.710052 2232 raft_consensus.cc:2270] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.710695 2232 ts_tablet_manager.cc:1905] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.711185 2106 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet 35c5be3d39c34774bcabfd66aaf8dbaa (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]) successfully deleted
I20250625 08:04:29.711331 2106 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet d25d775165474789bd7f87d39bdf5e5f (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:29.711448 2106 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet d25d775165474789bd7f87d39bdf5e5f (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:29.711484 1691 tablet_service.cc:1468] Processing CreateTablet for tablet a24a0ae1aa5c40e4b1a87428190b1932 (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.711741 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a24a0ae1aa5c40e4b1a87428190b1932. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.711447 2111 master_service.cc:948] ReplaceTablet: received request to replace tablet 9deb9ceefd014d0996dd65fefb10e659 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.714147 2222 tablet_bootstrap.cc:492] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.714704 2114 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet 9deb9ceefd014d0996dd65fefb10e659
I20250625 08:04:29.715096 2114 catalog_manager.cc:6652] ReplaceTablet: tablet 9deb9ceefd014d0996dd65fefb10e659 deleted and replaced by tablet 2a6db42100cc4b5ca65202e34514e966
I20250625 08:04:29.715610 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet 9deb9ceefd014d0996dd65fefb10e659 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a6db42100cc4b5ca65202e34514e966) from {username='slave'} at 127.0.0.1:44472
I20250625 08:04:29.716261 1828 tablet_service.cc:1468] Processing CreateTablet for tablet a24a0ae1aa5c40e4b1a87428190b1932 (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.716547 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a24a0ae1aa5c40e4b1a87428190b1932. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.719274 1963 tablet_service.cc:1468] Processing CreateTablet for tablet a24a0ae1aa5c40e4b1a87428190b1932 (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.719591 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a24a0ae1aa5c40e4b1a87428190b1932. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.720046 1690 tablet_service.cc:1468] Processing CreateTablet for tablet 626d783b9280477a8dd9ee5cc7f79237 (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.720305 1690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 626d783b9280477a8dd9ee5cc7f79237. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.721056 1962 tablet_service.cc:1468] Processing CreateTablet for tablet 626d783b9280477a8dd9ee5cc7f79237 (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.721298 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 626d783b9280477a8dd9ee5cc7f79237. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.721805 2223 tablet_bootstrap.cc:492] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.723073 2223 tablet_bootstrap.cc:654] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.724471 2222 tablet_bootstrap.cc:654] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.725572 1827 tablet_service.cc:1515] Processing DeleteTablet for tablet 9deb9ceefd014d0996dd65fefb10e659 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a6db42100cc4b5ca65202e34514e966) from {username='slave'} at 127.0.0.1:53818
I20250625 08:04:29.725977 2229 tablet_replica.cc:331] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:29.726264 1826 tablet_service.cc:1468] Processing CreateTablet for tablet 626d783b9280477a8dd9ee5cc7f79237 (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.726521 2229 raft_consensus.cc:2241] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.726632 2229 raft_consensus.cc:2270] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.727155 2229 ts_tablet_manager.cc:1905] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.729358 2229 ts_tablet_manager.cc:1918] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.729429 2229 log.cc:1199] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/9deb9ceefd014d0996dd65fefb10e659
I20250625 08:04:29.729640 2229 ts_tablet_manager.cc:1939] T 9deb9ceefd014d0996dd65fefb10e659 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:29.730432 1826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 626d783b9280477a8dd9ee5cc7f79237. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.730427 1445 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=0e43d85f40e545d9b31c5cb3e62f89a8]
I20250625 08:04:29.730674 1445 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 29) took 2104 ms. Trace:
I20250625 08:04:29.730782 1690 tablet_service.cc:1468] Processing CreateTablet for tablet b845a398124a4964b02c15bf2630585c (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.730728 1445 rpcz_store.cc:276] 0625 08:04:27.626361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626433 (+ 72us) service_pool.cc:225] Handling call
0625 08:04:28.180931 (+554498us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:28.181216 (+ 285us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.459222 (+278006us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.459810 (+ 588us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.459812 (+ 2us) write_op.cc:454] Released schema lock
0625 08:04:28.459944 (+ 132us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.730658 (+1270714us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.459761 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER)
0625 08:04:28.459770 (+ 9us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.459774 (+ 4us) write_op.cc:454] Released schema lock
0625 08:04:28.459777 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1400660,"Hive Metastore.run_cpu_time_us":428,"Hive Metastore.run_wall_time_us":147480,"mutex_wait_us":56,"child_traces":[["op",{}]]}
I20250625 08:04:29.731076 1690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b845a398124a4964b02c15bf2630585c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.731257 1689 tablet_service.cc:1468] Processing CreateTablet for tablet 2a6db42100cc4b5ca65202e34514e966 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.731518 1689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a6db42100cc4b5ca65202e34514e966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.732646 2221 tablet_bootstrap.cc:492] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.733687 2221 tablet_bootstrap.cc:654] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.735859 2111 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_f22b18eed5c34ffaa7af847a3c64598d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.736044 2106 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 9deb9ceefd014d0996dd65fefb10e659 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:29.736171 1827 tablet_service.cc:1468] Processing CreateTablet for tablet 2a6db42100cc4b5ca65202e34514e966 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.736243 2114 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_88079d64160a4318bf8ea1807cb3a30d" } modify_external_catalogs: true
I20250625 08:04:29.736371 2114 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_88079d64160a4318bf8ea1807cb3a30d" } modify_external_catalogs: true
I20250625 08:04:29.736433 1827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a6db42100cc4b5ca65202e34514e966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.739873 2221 tablet_bootstrap.cc:492] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.742378 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d25d775165474789bd7f87d39bdf5e5f" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
W20250625 08:04:29.742580 1647 leader_election.cc:343] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: d25d775165474789bd7f87d39bdf5e5f
W20250625 08:04:29.742990 1647 leader_election.cc:343] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Illegal state: Tablet not RUNNING: STOPPED
I20250625 08:04:29.743072 1647 leader_election.cc:304] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
W20250625 08:04:29.743268 1647 raft_consensus.cc:2684] T d25d775165474789bd7f87d39bdf5e5f P eebfa8472fe04080ac012e2c91bedaae: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 08:04:29.743357 2223 tablet_bootstrap.cc:492] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.743445 2223 ts_tablet_manager.cc:1397] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250625 08:04:29.743973 2223 raft_consensus.cc:357] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.744076 2223 raft_consensus.cc:383] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.744110 2223 raft_consensus.cc:738] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.744292 2223 consensus_queue.cc:260] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.746377 2223 ts_tablet_manager.cc:1428] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 08:04:29.747429 2223 tablet_bootstrap.cc:492] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.748781 2223 tablet_bootstrap.cc:654] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.749786 1962 tablet_service.cc:1468] Processing CreateTablet for tablet b845a398124a4964b02c15bf2630585c (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.750067 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b845a398124a4964b02c15bf2630585c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.750665 2221 ts_tablet_manager.cc:1397] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250625 08:04:29.751096 1691 tablet_service.cc:1515] Processing DeleteTablet for tablet 9deb9ceefd014d0996dd65fefb10e659 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a6db42100cc4b5ca65202e34514e966) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:29.752015 2231 tablet_replica.cc:331] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:29.752274 2231 raft_consensus.cc:2241] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.752434 2231 raft_consensus.cc:2270] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.753010 2231 ts_tablet_manager.cc:1905] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.753209 1826 tablet_service.cc:1468] Processing CreateTablet for tablet b845a398124a4964b02c15bf2630585c (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.753479 1826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b845a398124a4964b02c15bf2630585c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.755259 2221 raft_consensus.cc:357] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.755391 2221 raft_consensus.cc:383] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.755426 2221 raft_consensus.cc:738] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.755451 2231 ts_tablet_manager.cc:1918] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.755513 2231 log.cc:1199] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/9deb9ceefd014d0996dd65fefb10e659
I20250625 08:04:29.755571 2221 consensus_queue.cc:260] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.755723 2231 ts_tablet_manager.cc:1939] T 9deb9ceefd014d0996dd65fefb10e659 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:29.756489 2106 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 9deb9ceefd014d0996dd65fefb10e659 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:29.757051 2221 ts_tablet_manager.cc:1428] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250625 08:04:29.757247 2221 tablet_bootstrap.cc:492] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.758347 2221 tablet_bootstrap.cc:654] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.759209 1963 tablet_service.cc:1468] Processing CreateTablet for tablet 2a6db42100cc4b5ca65202e34514e966 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.759508 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a6db42100cc4b5ca65202e34514e966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.761664 2221 tablet_bootstrap.cc:492] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.761931 2221 ts_tablet_manager.cc:1397] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 08:04:29.762545 2221 raft_consensus.cc:357] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.762717 2221 raft_consensus.cc:383] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.762790 2221 raft_consensus.cc:738] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.762987 2221 consensus_queue.cc:260] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.763646 2223 tablet_bootstrap.cc:492] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.763743 2223 ts_tablet_manager.cc:1397] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20250625 08:04:29.764268 2223 raft_consensus.cc:357] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.764364 2223 raft_consensus.cc:383] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.764395 2223 raft_consensus.cc:738] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.764511 2223 consensus_queue.cc:260] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.764806 2223 ts_tablet_manager.cc:1428] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:29.764984 2223 tablet_bootstrap.cc:492] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.766048 2223 tablet_bootstrap.cc:654] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.768561 2232 ts_tablet_manager.cc:1918] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.768646 2232 log.cc:1199] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/d25d775165474789bd7f87d39bdf5e5f
I20250625 08:04:29.768851 2232 ts_tablet_manager.cc:1939] T d25d775165474789bd7f87d39bdf5e5f P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:29.769529 2232 tablet_replica.cc:331] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:29.769649 2232 raft_consensus.cc:2241] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:29.769798 2232 pending_rounds.cc:70] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: Trying to abort 1 pending ops.
I20250625 08:04:29.769826 2232 pending_rounds.cc:77] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171435190770868224 op_type: NO_OP noop_request { }
I20250625 08:04:29.769887 2232 raft_consensus.cc:2887] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 08:04:29.769958 2232 raft_consensus.cc:2270] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.774617 2112 catalog_manager.cc:2149] created HMS catalog entry for table [id=b4ad9fa057114ca4878623901a7a5206]
I20250625 08:04:29.779151 2222 tablet_bootstrap.cc:492] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.779285 2222 ts_tablet_manager.cc:1397] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.000s
I20250625 08:04:29.779809 2222 raft_consensus.cc:357] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.779930 2222 raft_consensus.cc:383] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.779963 2222 raft_consensus.cc:738] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.780092 2222 consensus_queue.cc:260] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.780355 2222 ts_tablet_manager.cc:1428] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 08:04:29.780591 2222 tablet_bootstrap.cc:492] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.781672 2222 tablet_bootstrap.cc:654] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.782832 2232 ts_tablet_manager.cc:1905] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.785166 2232 ts_tablet_manager.cc:1918] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:29.785260 2232 log.cc:1199] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/9deb9ceefd014d0996dd65fefb10e659
I20250625 08:04:29.785513 2232 ts_tablet_manager.cc:1939] T 9deb9ceefd014d0996dd65fefb10e659 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:29.786221 2112 catalog_manager.cc:5674] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071) does not have the latest schema for tablet 626d783b9280477a8dd9ee5cc7f79237 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
W20250625 08:04:29.786562 2112 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 626d783b9280477a8dd9ee5cc7f79237 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 626d783b9280477a8dd9ee5cc7f79237 consensus state has no leader
I20250625 08:04:29.787611 2221 ts_tablet_manager.cc:1428] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250625 08:04:29.795490 2221 tablet_bootstrap.cc:492] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.796530 2106 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet d25d775165474789bd7f87d39bdf5e5f (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:29.796725 2106 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet 9deb9ceefd014d0996dd65fefb10e659 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:29.797694 2221 tablet_bootstrap.cc:654] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.801529 2223 tablet_bootstrap.cc:492] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.801663 2223 ts_tablet_manager.cc:1397] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20250625 08:04:29.802287 2223 raft_consensus.cc:357] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.802414 2223 raft_consensus.cc:383] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.802475 2223 raft_consensus.cc:738] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.802606 1691 tablet_service.cc:1468] Processing CreateTablet for tablet c8549c4ed76541faad94532cd923745c (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.802620 2223 consensus_queue.cc:260] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.802934 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8549c4ed76541faad94532cd923745c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.802927 2223 ts_tablet_manager.cc:1428] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.803117 2223 tablet_bootstrap.cc:492] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.803290 2112 catalog_manager.cc:5674] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005) does not have the latest schema for tablet 626d783b9280477a8dd9ee5cc7f79237 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
I20250625 08:04:29.804302 2223 tablet_bootstrap.cc:654] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.804706 2112 catalog_manager.cc:5674] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071) does not have the latest schema for tablet 626d783b9280477a8dd9ee5cc7f79237 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
I20250625 08:04:29.805123 1826 tablet_service.cc:1468] Processing CreateTablet for tablet c8549c4ed76541faad94532cd923745c (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.805341 2233 raft_consensus.cc:491] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:29.805428 1826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8549c4ed76541faad94532cd923745c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.805424 2233 raft_consensus.cc:513] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.805444 2112 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_d78071777f3646548f7f75cea674ecdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:29.805836 2233 leader_election.cc:290] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.806205 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a24a0ae1aa5c40e4b1a87428190b1932" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:29.806322 1982 raft_consensus.cc:2466] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:29.806295 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a24a0ae1aa5c40e4b1a87428190b1932" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
W20250625 08:04:29.807000 1647 leader_election.cc:343] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:04:29.807152 1647 leader_election.cc:304] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters: de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:29.807324 2233 raft_consensus.cc:2802] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:29.807384 2233 raft_consensus.cc:491] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:29.807430 2233 raft_consensus.cc:3058] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.807976 1963 tablet_service.cc:1468] Processing CreateTablet for tablet c8549c4ed76541faad94532cd923745c (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.808259 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8549c4ed76541faad94532cd923745c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.808421 2233 raft_consensus.cc:513] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.808761 2233 leader_election.cc:290] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.809700 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a24a0ae1aa5c40e4b1a87428190b1932" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:29.809808 1982 raft_consensus.cc:3058] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.810690 1982 raft_consensus.cc:2466] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:29.811024 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a24a0ae1aa5c40e4b1a87428190b1932" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:29.811568 1647 leader_election.cc:304] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
W20250625 08:04:29.811760 1647 leader_election.cc:343] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:04:29.811983 2233 raft_consensus.cc:2802] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:29.812243 2233 raft_consensus.cc:695] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:29.812558 2233 consensus_queue.cc:237] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.814332 2222 tablet_bootstrap.cc:492] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.814472 2222 ts_tablet_manager.cc:1397] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20250625 08:04:29.814790 2221 tablet_bootstrap.cc:492] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.814898 2221 ts_tablet_manager.cc:1397] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20250625 08:04:29.815011 2222 raft_consensus.cc:357] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.815116 2222 raft_consensus.cc:383] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.815151 2222 raft_consensus.cc:738] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.815266 2222 consensus_queue.cc:260] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.815374 2221 raft_consensus.cc:357] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.815487 2221 raft_consensus.cc:383] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.815521 2221 raft_consensus.cc:738] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.815537 2222 ts_tablet_manager.cc:1428] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.815639 2221 consensus_queue.cc:260] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.815714 2222 tablet_bootstrap.cc:492] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.815896 2221 ts_tablet_manager.cc:1428] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.816107 2221 tablet_bootstrap.cc:492] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.816804 2222 tablet_bootstrap.cc:654] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.817106 2221 tablet_bootstrap.cc:654] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.819824 2222 tablet_bootstrap.cc:492] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.819922 2222 ts_tablet_manager.cc:1397] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 08:04:29.820389 2222 raft_consensus.cc:357] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.820482 2222 raft_consensus.cc:383] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.820508 2222 raft_consensus.cc:738] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.820611 2222 consensus_queue.cc:260] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.820828 2222 ts_tablet_manager.cc:1428] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.821005 2222 tablet_bootstrap.cc:492] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.822274 2222 tablet_bootstrap.cc:654] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.823220 2221 tablet_bootstrap.cc:492] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.823329 2221 ts_tablet_manager.cc:1397] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250625 08:04:29.823855 2221 raft_consensus.cc:357] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.823966 2221 raft_consensus.cc:383] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.823995 2221 raft_consensus.cc:738] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.824144 2221 consensus_queue.cc:260] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.825825 2222 tablet_bootstrap.cc:492] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.825918 2222 ts_tablet_manager.cc:1397] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 08:04:29.826318 2221 ts_tablet_manager.cc:1428] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 08:04:29.826345 2222 raft_consensus.cc:357] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.826431 2222 raft_consensus.cc:383] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.826972 2222 raft_consensus.cc:738] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.827095 2222 consensus_queue.cc:260] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.827347 2222 ts_tablet_manager.cc:1428] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.827513 2222 tablet_bootstrap.cc:492] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.828521 2222 tablet_bootstrap.cc:654] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.829988 2221 tablet_bootstrap.cc:492] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.832631 2222 tablet_bootstrap.cc:492] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.832736 2222 ts_tablet_manager.cc:1397] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 08:04:29.832736 2221 tablet_bootstrap.cc:654] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.833191 2222 raft_consensus.cc:357] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:29.833297 2222 raft_consensus.cc:383] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.833325 2222 raft_consensus.cc:738] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.833443 2222 consensus_queue.cc:260] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:29.833634 2243 raft_consensus.cc:491] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:29.833691 2222 ts_tablet_manager.cc:1428] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.833732 2243 raft_consensus.cc:513] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.834121 2243 leader_election.cc:290] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:29.835712 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b845a398124a4964b02c15bf2630585c" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:29.835855 1982 raft_consensus.cc:2466] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:29.836218 2223 tablet_bootstrap.cc:492] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.836295 1784 leader_election.cc:304] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:29.836324 2223 ts_tablet_manager.cc:1397] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20250625 08:04:29.836445 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b845a398124a4964b02c15bf2630585c" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:29.836549 2243 raft_consensus.cc:2802] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:29.836599 2243 raft_consensus.cc:491] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:29.836583 1711 raft_consensus.cc:2466] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:29.836637 2243 raft_consensus.cc:3058] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.836894 2223 raft_consensus.cc:357] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.837013 2223 raft_consensus.cc:383] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.837044 2223 raft_consensus.cc:738] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.837165 2223 consensus_queue.cc:260] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.837545 2223 ts_tablet_manager.cc:1428] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:29.837702 2243 raft_consensus.cc:513] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.837819 2223 tablet_bootstrap.cc:492] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.838093 2243 leader_election.cc:290] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:29.839360 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b845a398124a4964b02c15bf2630585c" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:29.839496 1711 raft_consensus.cc:3058] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.840585 2223 tablet_bootstrap.cc:654] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.840572 1711 raft_consensus.cc:2466] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:29.841120 1784 leader_election.cc:304] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:29.841380 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b845a398124a4964b02c15bf2630585c" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:29.841502 1982 raft_consensus.cc:3058] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.842550 1982 raft_consensus.cc:2466] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:29.843127 2243 raft_consensus.cc:2802] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:29.843329 2243 raft_consensus.cc:695] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:29.843475 2243 consensus_queue.cc:237] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.843915 1443 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=d2b3f7fc195b480f9e7d191a8ed99599]
I20250625 08:04:29.844139 1443 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 25) took 2218 ms. Trace:
I20250625 08:04:29.844182 1443 rpcz_store.cc:276] 0625 08:04:27.626046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626132 (+ 86us) service_pool.cc:225] Handling call
0625 08:04:28.181560 (+555428us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:28.181855 (+ 295us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.481823 (+299968us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.482467 (+ 644us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.482469 (+ 2us) write_op.cc:454] Released schema lock
0625 08:04:28.482594 (+ 125us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.844124 (+1361530us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.482383 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER)
0625 08:04:28.482391 (+ 8us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.482397 (+ 6us) write_op.cc:454] Released schema lock
0625 08:04:28.482400 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1524963,"Hive Metastore.run_cpu_time_us":421,"Hive Metastore.run_wall_time_us":136041,"mutex_wait_us":82,"child_traces":[["op",{}]]}
I20250625 08:04:29.849144 2221 tablet_bootstrap.cc:492] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.849406 2221 ts_tablet_manager.cc:1397] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.003s
I20250625 08:04:29.849993 2221 raft_consensus.cc:357] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:29.850207 2221 raft_consensus.cc:383] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.850282 2221 raft_consensus.cc:738] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.850519 2221 consensus_queue.cc:260] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:29.853274 2221 ts_tablet_manager.cc:1428] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 08:04:29.857671 2223 tablet_bootstrap.cc:492] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.857795 2223 ts_tablet_manager.cc:1397] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250625 08:04:29.858377 2223 raft_consensus.cc:357] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:29.858655 2223 raft_consensus.cc:383] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.858695 2223 raft_consensus.cc:738] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.858850 2223 consensus_queue.cc:260] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:29.859135 2223 ts_tablet_manager.cc:1428] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.874800 2256 raft_consensus.cc:491] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:29.874958 2256 raft_consensus.cc:513] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.875496 2256 leader_election.cc:290] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.876338 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "626d783b9280477a8dd9ee5cc7f79237" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:29.876363 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "626d783b9280477a8dd9ee5cc7f79237" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:29.876507 1847 raft_consensus.cc:2466] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:29.876528 1711 raft_consensus.cc:2466] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:29.876971 1919 leader_election.cc:304] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:29.877308 2256 raft_consensus.cc:2802] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:29.877377 2256 raft_consensus.cc:491] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:29.877424 2256 raft_consensus.cc:3058] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.878628 2256 raft_consensus.cc:513] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.879060 2256 leader_election.cc:290] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.879632 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "626d783b9280477a8dd9ee5cc7f79237" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:29.879631 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "626d783b9280477a8dd9ee5cc7f79237" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:29.879765 1711 raft_consensus.cc:3058] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.879768 1847 raft_consensus.cc:3058] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.880786 1711 raft_consensus.cc:2466] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:29.880897 1847 raft_consensus.cc:2466] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:29.881182 1919 leader_election.cc:304] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:29.881362 2256 raft_consensus.cc:2802] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:29.881449 2256 raft_consensus.cc:695] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:29.881611 2256 consensus_queue.cc:237] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.889645 2113 catalog_manager.cc:2149] created HMS catalog entry for table [id=874b84d5e2004697a01c0686260a6904]
I20250625 08:04:29.896952 2113 catalog_manager.cc:5582] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae reported cstate change: term changed from 0 to 1, leader changed from <none> to eebfa8472fe04080ac012e2c91bedaae (127.1.19.65). New cstate: current_term: 1 leader_uuid: "eebfa8472fe04080ac012e2c91bedaae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } health_report { overall_health: UNKNOWN } } }
I20250625 08:04:29.911079 2113 catalog_manager.cc:5582] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67). New cstate: current_term: 1 leader_uuid: "5dae8595b9484b57bc7ccea543f83130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } health_report { overall_health: UNKNOWN } } }
I20250625 08:04:29.911350 2113 catalog_manager.cc:5674] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005) does not have the latest schema for tablet 626d783b9280477a8dd9ee5cc7f79237 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
I20250625 08:04:29.912081 1826 tablet_service.cc:1468] Processing CreateTablet for tablet 7342ad43df3d493b994075493c6ab8a4 (DEFAULT_TABLE table=default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.912420 1826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7342ad43df3d493b994075493c6ab8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.912710 1691 tablet_service.cc:1468] Processing CreateTablet for tablet 7342ad43df3d493b994075493c6ab8a4 (DEFAULT_TABLE table=default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.913002 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7342ad43df3d493b994075493c6ab8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.914945 1963 tablet_service.cc:1468] Processing CreateTablet for tablet 7342ad43df3d493b994075493c6ab8a4 (DEFAULT_TABLE table=default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.915201 2222 tablet_bootstrap.cc:492] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.915225 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7342ad43df3d493b994075493c6ab8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.916235 2222 tablet_bootstrap.cc:654] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.917594 2221 tablet_bootstrap.cc:492] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.918577 2113 catalog_manager.cc:5674] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085) does not have the latest schema for tablet 626d783b9280477a8dd9ee5cc7f79237 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]). Expected version 1 got 0
I20250625 08:04:29.918648 2221 tablet_bootstrap.cc:654] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.919773 2223 tablet_bootstrap.cc:492] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.919729 2113 catalog_manager.cc:5582] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf reported cstate change: term changed from 0 to 1, leader changed from <none> to de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66). New cstate: current_term: 1 leader_uuid: "de62034f80d9463dbd6386fc3e90fdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } health_report { overall_health: HEALTHY } } }
I20250625 08:04:29.919894 2221 tablet_bootstrap.cc:492] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.919976 2221 ts_tablet_manager.cc:1397] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:29.920337 2116 catalog_manager.cc:2149] created HMS catalog entry for table [id=8226dc4b416b4b458db9460ca40cd64d]
I20250625 08:04:29.920464 2221 raft_consensus.cc:357] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.920564 2221 raft_consensus.cc:383] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.920598 2221 raft_consensus.cc:738] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.920722 2221 consensus_queue.cc:260] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.920842 2223 tablet_bootstrap.cc:654] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.921015 2221 ts_tablet_manager.cc:1428] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:29.925743 2116 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:29.925882 2116 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:29.928529 2222 tablet_bootstrap.cc:492] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.928646 2222 ts_tablet_manager.cc:1397] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 08:04:29.929199 2222 raft_consensus.cc:357] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.929333 2222 raft_consensus.cc:383] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.930027 1438 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table [id=b53c21c44e124643927d73a514201a8e]
I20250625 08:04:29.930222 1438 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35718 (request call id 34) took 2303 ms. Trace:
I20250625 08:04:29.930269 1438 rpcz_store.cc:276] 0625 08:04:27.626778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 08:04:27.626843 (+ 65us) service_pool.cc:225] Handling call
0625 08:04:28.182190 (+555347us) catalog_manager.cc:2076] Acquired catalog manager lock
0625 08:04:28.182502 (+ 312us) catalog_manager.cc:2122] Created new table and tablet info
0625 08:04:28.504857 (+322355us) catalog_manager.cc:2148] Created new table in HMS catalog
0625 08:04:28.505478 (+ 621us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.505480 (+ 2us) write_op.cc:454] Released schema lock
0625 08:04:28.505609 (+ 129us) catalog_manager.cc:2156] Rolling back HMS table creation
0625 08:04:29.930213 (+1424604us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 08:04:28.505429 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 0404561252194ce29bf2924fd5e67e22 is not leader of this config: current role FOLLOWER)
0625 08:04:28.505438 (+ 9us) write_op.cc:489] Releasing partition, row and schema locks
0625 08:04:28.505443 (+ 5us) write_op.cc:454] Released schema lock
0625 08:04:28.505446 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1637448,"Hive Metastore.run_cpu_time_us":438,"Hive Metastore.run_wall_time_us":109053,"mutex_wait_us":153,"spinlock_wait_cycles":19328,"child_traces":[["op",{}]]}
I20250625 08:04:29.934314 2113 master_service.cc:948] ReplaceTablet: received request to replace tablet a24a0ae1aa5c40e4b1a87428190b1932 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.929365 2222 raft_consensus.cc:738] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.936069 2222 consensus_queue.cc:260] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.937994 2223 tablet_bootstrap.cc:492] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.938117 2223 ts_tablet_manager.cc:1397] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250625 08:04:29.939225 2223 raft_consensus.cc:357] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.939352 2223 raft_consensus.cc:383] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.939393 2223 raft_consensus.cc:738] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.939388 2222 ts_tablet_manager.cc:1428] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 08:04:29.939523 2223 consensus_queue.cc:260] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:29.939783 2223 ts_tablet_manager.cc:1428] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 08:04:29.944545 2113 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet a24a0ae1aa5c40e4b1a87428190b1932
I20250625 08:04:29.944829 2113 catalog_manager.cc:6652] ReplaceTablet: tablet a24a0ae1aa5c40e4b1a87428190b1932 deleted and replaced by tablet 579f822528014659a13b9dc93111624f
I20250625 08:04:29.944911 1963 tablet_service.cc:1468] Processing CreateTablet for tablet bf12ce8b7b43487aa8bfb6b1e9f88816 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.945187 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf12ce8b7b43487aa8bfb6b1e9f88816. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.946851 2113 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:29.946982 2113 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57934:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:29.948931 1828 tablet_service.cc:1515] Processing DeleteTablet for tablet a24a0ae1aa5c40e4b1a87428190b1932 with delete_type TABLET_DATA_DELETED (replaced by tablet 579f822528014659a13b9dc93111624f) from {username='slave'} at 127.0.0.1:53818
I20250625 08:04:29.949061 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet a24a0ae1aa5c40e4b1a87428190b1932 with delete_type TABLET_DATA_DELETED (replaced by tablet 579f822528014659a13b9dc93111624f) from {username='slave'} at 127.0.0.1:44472
I20250625 08:04:29.949148 2229 tablet_replica.cc:331] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:29.949225 2232 tablet_replica.cc:331] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:29.949295 2229 raft_consensus.cc:2241] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.949337 2232 raft_consensus.cc:2241] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:29.949419 2229 raft_consensus.cc:2270] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.949452 2232 raft_consensus.cc:2270] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.950083 2232 ts_tablet_manager.cc:1905] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.950170 1689 tablet_service.cc:1515] Processing DeleteTablet for tablet a24a0ae1aa5c40e4b1a87428190b1932 with delete_type TABLET_DATA_DELETED (replaced by tablet 579f822528014659a13b9dc93111624f) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:29.950278 1826 tablet_service.cc:1468] Processing CreateTablet for tablet bf12ce8b7b43487aa8bfb6b1e9f88816 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.950352 2231 tablet_replica.cc:331] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:29.950493 2231 raft_consensus.cc:2241] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:29.950570 1826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf12ce8b7b43487aa8bfb6b1e9f88816. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.950764 2231 pending_rounds.cc:70] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: Trying to abort 1 pending ops.
I20250625 08:04:29.950798 2229 ts_tablet_manager.cc:1905] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.950887 2231 pending_rounds.cc:77] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171435191555764224 op_type: NO_OP noop_request { }
I20250625 08:04:29.950989 2231 raft_consensus.cc:2887] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 08:04:29.951051 2231 raft_consensus.cc:2270] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:29.952412 2221 tablet_bootstrap.cc:492] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.952854 1691 tablet_service.cc:1468] Processing CreateTablet for tablet bf12ce8b7b43487aa8bfb6b1e9f88816 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.953049 2229 ts_tablet_manager.cc:1918] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.953115 2229 log.cc:1199] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/a24a0ae1aa5c40e4b1a87428190b1932
I20250625 08:04:29.953159 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf12ce8b7b43487aa8bfb6b1e9f88816. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.953326 2229 ts_tablet_manager.cc:1939] T a24a0ae1aa5c40e4b1a87428190b1932 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:29.953557 2221 tablet_bootstrap.cc:654] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.954712 2231 ts_tablet_manager.cc:1905] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:29.956807 2231 ts_tablet_manager.cc:1918] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:29.956880 2231 log.cc:1199] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/a24a0ae1aa5c40e4b1a87428190b1932
I20250625 08:04:29.957152 2231 ts_tablet_manager.cc:1939] T a24a0ae1aa5c40e4b1a87428190b1932 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:29.957607 2106 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet a24a0ae1aa5c40e4b1a87428190b1932 (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]) successfully deleted
I20250625 08:04:29.957826 2232 ts_tablet_manager.cc:1918] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:29.957947 2232 log.cc:1199] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/a24a0ae1aa5c40e4b1a87428190b1932
I20250625 08:04:29.958163 2232 ts_tablet_manager.cc:1939] T a24a0ae1aa5c40e4b1a87428190b1932 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:29.959908 2222 tablet_bootstrap.cc:492] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.959928 2223 tablet_bootstrap.cc:492] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.960852 2106 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet a24a0ae1aa5c40e4b1a87428190b1932 (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]) successfully deleted
I20250625 08:04:29.961071 2106 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet a24a0ae1aa5c40e4b1a87428190b1932 (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]) successfully deleted
I20250625 08:04:29.961225 1828 tablet_service.cc:1468] Processing CreateTablet for tablet 579f822528014659a13b9dc93111624f (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.961508 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 579f822528014659a13b9dc93111624f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.962699 1963 tablet_service.cc:1468] Processing CreateTablet for tablet 579f822528014659a13b9dc93111624f (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.963088 2222 tablet_bootstrap.cc:654] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.963791 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 579f822528014659a13b9dc93111624f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.964212 2222 tablet_bootstrap.cc:492] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.964352 2222 ts_tablet_manager.cc:1397] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 08:04:29.964948 2222 raft_consensus.cc:357] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.965147 2222 raft_consensus.cc:383] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.965235 2222 raft_consensus.cc:738] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.965318 1691 tablet_service.cc:1468] Processing CreateTablet for tablet 579f822528014659a13b9dc93111624f (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:29.965489 2222 consensus_queue.cc:260] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.965593 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 579f822528014659a13b9dc93111624f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:29.965867 2223 tablet_bootstrap.cc:654] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.966831 2221 tablet_bootstrap.cc:492] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:29.966921 2221 ts_tablet_manager.cc:1397] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20250625 08:04:29.967417 2221 raft_consensus.cc:357] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.967525 2221 raft_consensus.cc:383] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.967562 2221 raft_consensus.cc:738] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.967681 2221 consensus_queue.cc:260] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.967984 2221 ts_tablet_manager.cc:1428] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:29.968185 2221 tablet_bootstrap.cc:492] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:29.969219 2221 tablet_bootstrap.cc:654] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.971766 2222 ts_tablet_manager.cc:1428] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250625 08:04:29.972008 2222 tablet_bootstrap.cc:492] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:29.973513 2222 tablet_bootstrap.cc:654] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:29.974630 2222 tablet_bootstrap.cc:492] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:29.974776 2222 ts_tablet_manager.cc:1397] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:29.975334 2222 raft_consensus.cc:357] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.975538 2222 raft_consensus.cc:383] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.975620 2222 raft_consensus.cc:738] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.975823 2222 consensus_queue.cc:260] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.976611 2222 ts_tablet_manager.cc:1428] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 08:04:29.981675 2117 catalog_manager.cc:2149] created HMS catalog entry for table [id=c0c4794ff9864a07a9f6322526e4397a]
I20250625 08:04:29.984890 2223 tablet_bootstrap.cc:492] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:29.985040 2223 ts_tablet_manager.cc:1397] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20250625 08:04:29.985577 2223 raft_consensus.cc:357] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.985706 2223 raft_consensus.cc:383] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:29.985744 2223 raft_consensus.cc:738] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:29.985870 2223 consensus_queue.cc:260] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.986153 2223 ts_tablet_manager.cc:1428] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250625 08:04:29.986382 2223 tablet_bootstrap.cc:492] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:29.989250 2117 master_service.cc:948] ReplaceTablet: received request to replace tablet b845a398124a4964b02c15bf2630585c from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:29.989409 2256 raft_consensus.cc:491] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:29.989507 2256 raft_consensus.cc:513] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.989864 2256 leader_election.cc:290] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.990806 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a6db42100cc4b5ca65202e34514e966" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:29.990968 1711 raft_consensus.cc:2466] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:29.991207 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a6db42100cc4b5ca65202e34514e966" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:29.991335 1847 raft_consensus.cc:2466] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:29.992931 1919 leader_election.cc:304] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:29.995141 2256 raft_consensus.cc:2802] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:29.995229 2256 raft_consensus.cc:491] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:29.995271 2256 raft_consensus.cc:3058] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.995986 2270 raft_consensus.cc:491] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:29.996093 2270 raft_consensus.cc:513] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:29.996372 2256 raft_consensus.cc:513] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:29.996563 2270 leader_election.cc:290] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:29.996774 2256 leader_election.cc:290] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:29.998258 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a6db42100cc4b5ca65202e34514e966" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:29.998395 1847 raft_consensus.cc:3058] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.998670 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8549c4ed76541faad94532cd923745c" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:29.998772 1848 raft_consensus.cc:2466] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:29.999127 1647 leader_election.cc:304] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:29.999313 2270 raft_consensus.cc:2802] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:29.999369 2270 raft_consensus.cc:491] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:29.999408 2270 raft_consensus.cc:3058] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:29.999498 1847 raft_consensus.cc:2466] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:29.999987 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a6db42100cc4b5ca65202e34514e966" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:30.000108 1711 raft_consensus.cc:3058] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.000475 2270 raft_consensus.cc:513] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:30.000851 2270 leader_election.cc:290] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:30.001037 1711 raft_consensus.cc:2466] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:30.001305 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8549c4ed76541faad94532cd923745c" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:30.001420 1847 raft_consensus.cc:3058] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.001714 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8549c4ed76541faad94532cd923745c" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:30.001840 1982 raft_consensus.cc:2466] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:30.002076 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8549c4ed76541faad94532cd923745c" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:30.002166 1983 raft_consensus.cc:3058] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.002338 1847 raft_consensus.cc:2466] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:30.002803 1691 tablet_service.cc:1468] Processing CreateTablet for tablet 40af5067b7224accb99733af91d6ca85 (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:30.003091 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40af5067b7224accb99733af91d6ca85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:30.003219 1983 raft_consensus.cc:2466] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:30.003610 1828 tablet_service.cc:1468] Processing CreateTablet for tablet 40af5067b7224accb99733af91d6ca85 (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:30.003866 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40af5067b7224accb99733af91d6ca85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:30.004405 2117 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet b845a398124a4964b02c15bf2630585c
I20250625 08:04:30.005052 2117 catalog_manager.cc:6652] ReplaceTablet: tablet b845a398124a4964b02c15bf2630585c deleted and replaced by tablet 17e0682e2bfe43b7a445f595d8929034
I20250625 08:04:30.005067 1689 tablet_service.cc:1515] Processing DeleteTablet for tablet b845a398124a4964b02c15bf2630585c with delete_type TABLET_DATA_DELETED (replaced by tablet 17e0682e2bfe43b7a445f595d8929034) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:30.005451 1826 tablet_service.cc:1515] Processing DeleteTablet for tablet b845a398124a4964b02c15bf2630585c with delete_type TABLET_DATA_DELETED (replaced by tablet 17e0682e2bfe43b7a445f595d8929034) from {username='slave'} at 127.0.0.1:53818
I20250625 08:04:30.005803 2222 tablet_bootstrap.cc:492] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:30.006032 2231 tablet_replica.cc:331] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:30.006160 2231 raft_consensus.cc:2241] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:30.006259 2231 raft_consensus.cc:2270] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:30.006414 2229 tablet_replica.cc:331] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:30.006559 2229 raft_consensus.cc:2241] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:30.006702 2229 pending_rounds.cc:70] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: Trying to abort 1 pending ops.
I20250625 08:04:30.006728 2229 pending_rounds.cc:77] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171435191684386816 op_type: NO_OP noop_request { }
I20250625 08:04:30.006788 2229 raft_consensus.cc:2887] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 08:04:30.006834 2229 raft_consensus.cc:2270] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:30.007215 2222 tablet_bootstrap.cc:654] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.007437 2231 ts_tablet_manager.cc:1905] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:30.009630 2231 ts_tablet_manager.cc:1918] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:30.009718 2231 log.cc:1199] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/b845a398124a4964b02c15bf2630585c
I20250625 08:04:30.009930 2231 ts_tablet_manager.cc:1939] T b845a398124a4964b02c15bf2630585c P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:30.011961 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet b845a398124a4964b02c15bf2630585c with delete_type TABLET_DATA_DELETED (replaced by tablet 17e0682e2bfe43b7a445f595d8929034) from {username='slave'} at 127.0.0.1:44472
I20250625 08:04:30.012166 2232 tablet_replica.cc:331] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:30.012288 2232 raft_consensus.cc:2241] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:30.012305 2229 ts_tablet_manager.cc:1905] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:30.012398 2232 raft_consensus.cc:2270] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:30.012498 2222 tablet_bootstrap.cc:492] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:30.012601 2222 ts_tablet_manager.cc:1397] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250625 08:04:30.013140 2222 raft_consensus.cc:357] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.013262 2222 raft_consensus.cc:383] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.013293 2222 raft_consensus.cc:738] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.013418 2222 consensus_queue.cc:260] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.013697 2222 ts_tablet_manager.cc:1428] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 08:04:30.013852 2270 raft_consensus.cc:491] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:30.013929 2270 raft_consensus.cc:513] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.014252 2270 leader_election.cc:290] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:30.014811 1919 leader_election.cc:304] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:30.014961 2232 ts_tablet_manager.cc:1905] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:30.015293 2229 ts_tablet_manager.cc:1918] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:30.015373 2229 log.cc:1199] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/b845a398124a4964b02c15bf2630585c
I20250625 08:04:30.015527 1647 leader_election.cc:304] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:30.015633 2229 ts_tablet_manager.cc:1939] T b845a398124a4964b02c15bf2630585c P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:30.016476 2233 raft_consensus.cc:2802] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:30.016562 2233 raft_consensus.cc:695] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:30.016682 2233 consensus_queue.cc:237] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:30.017059 1691 tablet_service.cc:1468] Processing CreateTablet for tablet 17e0682e2bfe43b7a445f595d8929034 (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:30.017336 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e0682e2bfe43b7a445f595d8929034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:30.018731 2106 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet b845a398124a4964b02c15bf2630585c (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]) successfully deleted
I20250625 08:04:30.018774 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40af5067b7224accb99733af91d6ca85" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:30.019752 2222 tablet_bootstrap.cc:492] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:30.019860 1962 tablet_service.cc:1468] Processing CreateTablet for tablet 17e0682e2bfe43b7a445f595d8929034 (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:30.020905 2222 tablet_bootstrap.cc:654] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.021140 1963 tablet_service.cc:1468] Processing CreateTablet for tablet 40af5067b7224accb99733af91d6ca85 (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:30.021466 2117 catalog_manager.cc:5582] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae reported cstate change: term changed from 0 to 1, leader changed from <none> to eebfa8472fe04080ac012e2c91bedaae (127.1.19.65). New cstate: current_term: 1 leader_uuid: "eebfa8472fe04080ac012e2c91bedaae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } health_report { overall_health: HEALTHY } } }
W20250625 08:04:30.021559 1647 leader_election.cc:343] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 40af5067b7224accb99733af91d6ca85
W20250625 08:04:30.021688 1647 leader_election.cc:343] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:04:30.021739 1647 leader_election.cc:304] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:30.022013 2270 raft_consensus.cc:2747] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 08:04:30.023514 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e0682e2bfe43b7a445f595d8929034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:30.024585 2232 ts_tablet_manager.cc:1918] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:30.024655 2232 log.cc:1199] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/b845a398124a4964b02c15bf2630585c
I20250625 08:04:30.024863 2232 ts_tablet_manager.cc:1939] T b845a398124a4964b02c15bf2630585c P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:30.025709 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40af5067b7224accb99733af91d6ca85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:30.025722 2106 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet b845a398124a4964b02c15bf2630585c (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]) successfully deleted
I20250625 08:04:30.029320 2223 tablet_bootstrap.cc:654] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.030032 2117 master_service.cc:948] ReplaceTablet: received request to replace tablet 2a6db42100cc4b5ca65202e34514e966 from {username='slave'} at 127.0.0.1:57934
I20250625 08:04:30.030525 2256 raft_consensus.cc:2802] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:30.030618 2256 raft_consensus.cc:695] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:30.030743 2256 consensus_queue.cc:237] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.035252 1828 tablet_service.cc:1468] Processing CreateTablet for tablet 17e0682e2bfe43b7a445f595d8929034 (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:30.035567 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e0682e2bfe43b7a445f595d8929034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:30.036688 2117 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet 2a6db42100cc4b5ca65202e34514e966
I20250625 08:04:30.037235 1963 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a6db42100cc4b5ca65202e34514e966 with delete_type TABLET_DATA_DELETED (replaced by tablet 99a2e60d7c7a4c419b609aa64d8b7334) from {username='slave'} at 127.0.0.1:44472
I20250625 08:04:30.037451 2232 tablet_replica.cc:331] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:30.037590 2232 raft_consensus.cc:2241] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:30.037742 2232 pending_rounds.cc:70] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: Trying to abort 1 pending ops.
I20250625 08:04:30.037775 2232 pending_rounds.cc:77] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171435192447356928 op_type: NO_OP noop_request { }
I20250625 08:04:30.037842 2232 raft_consensus.cc:2887] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 08:04:30.037916 2232 raft_consensus.cc:2270] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:30.038125 1691 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a6db42100cc4b5ca65202e34514e966 with delete_type TABLET_DATA_DELETED (replaced by tablet 99a2e60d7c7a4c419b609aa64d8b7334) from {username='slave'} at 127.0.0.1:45806
I20250625 08:04:30.039505 1826 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a6db42100cc4b5ca65202e34514e966 with delete_type TABLET_DATA_DELETED (replaced by tablet 99a2e60d7c7a4c419b609aa64d8b7334) from {username='slave'} at 127.0.0.1:53818
I20250625 08:04:30.039685 2229 tablet_replica.cc:331] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:30.039796 2229 raft_consensus.cc:2241] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:30.039906 2229 raft_consensus.cc:2270] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:30.041271 2231 tablet_replica.cc:331] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:30.041428 2231 raft_consensus.cc:2241] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:30.041549 2231 raft_consensus.cc:2270] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:30.041724 2106 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet b845a398124a4964b02c15bf2630585c (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]) successfully deleted
I20250625 08:04:30.042155 2231 ts_tablet_manager.cc:1905] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:30.043033 2229 ts_tablet_manager.cc:1905] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:30.043330 2221 tablet_bootstrap.cc:492] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:30.043419 2221 ts_tablet_manager.cc:1397] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.000s
I20250625 08:04:30.043859 2221 raft_consensus.cc:357] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.043998 2221 raft_consensus.cc:383] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.044046 2221 raft_consensus.cc:738] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.044245 2221 consensus_queue.cc:260] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.044605 2221 ts_tablet_manager.cc:1428] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:30.044816 2221 tablet_bootstrap.cc:492] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:30.045047 2231 ts_tablet_manager.cc:1918] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:30.045123 2229 ts_tablet_manager.cc:1918] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 08:04:30.045203 2229 log.cc:1199] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/2a6db42100cc4b5ca65202e34514e966
I20250625 08:04:30.045168 2231 log.cc:1199] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/2a6db42100cc4b5ca65202e34514e966
I20250625 08:04:30.045433 2229 ts_tablet_manager.cc:1939] T 2a6db42100cc4b5ca65202e34514e966 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:30.045565 2231 ts_tablet_manager.cc:1939] T 2a6db42100cc4b5ca65202e34514e966 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:30.045949 2221 tablet_bootstrap.cc:654] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.046023 2106 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 2a6db42100cc4b5ca65202e34514e966 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:30.046588 2106 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 2a6db42100cc4b5ca65202e34514e966 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:30.047112 1963 tablet_service.cc:1468] Processing CreateTablet for tablet 99a2e60d7c7a4c419b609aa64d8b7334 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:30.047418 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99a2e60d7c7a4c419b609aa64d8b7334. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 08:04:30.048509 2294 consensus_queue.cc:1162] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5dae8595b9484b57bc7ccea543f83130" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250625 08:04:30.049005 2223 tablet_bootstrap.cc:492] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:30.049096 2223 ts_tablet_manager.cc:1397] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.000s
I20250625 08:04:30.049543 2223 raft_consensus.cc:357] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.049649 2223 raft_consensus.cc:383] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.049685 2223 raft_consensus.cc:738] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.049796 2223 consensus_queue.cc:260] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.050081 2223 ts_tablet_manager.cc:1428] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:30.050268 2223 tablet_bootstrap.cc:492] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:30.051088 1691 tablet_service.cc:1468] Processing CreateTablet for tablet 99a2e60d7c7a4c419b609aa64d8b7334 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:30.051411 2223 tablet_bootstrap.cc:654] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.051427 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99a2e60d7c7a4c419b609aa64d8b7334. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:30.051527 2117 catalog_manager.cc:6652] ReplaceTablet: tablet 2a6db42100cc4b5ca65202e34514e966 deleted and replaced by tablet 99a2e60d7c7a4c419b609aa64d8b7334
I20250625 08:04:30.055629 2117 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57934:
name: "default.Table_28db8798a1444ed2b1d46a5a3eba537b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:30.057713 2223 tablet_bootstrap.cc:492] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:30.057729 2221 tablet_bootstrap.cc:492] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:30.057825 2221 ts_tablet_manager.cc:1397] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 08:04:30.057825 2223 ts_tablet_manager.cc:1397] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250625 08:04:30.058354 2223 raft_consensus.cc:357] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.058354 2221 raft_consensus.cc:357] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.058506 2223 raft_consensus.cc:383] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.058516 2221 raft_consensus.cc:383] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.058560 2221 raft_consensus.cc:738] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.058560 2223 raft_consensus.cc:738] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.058727 2223 consensus_queue.cc:260] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.058849 2221 consensus_queue.cc:260] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.059054 2223 ts_tablet_manager.cc:1428] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:30.059401 2223 tablet_bootstrap.cc:492] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:30.060540 2223 tablet_bootstrap.cc:654] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.062605 2278 raft_consensus.cc:491] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:30.062721 2278 raft_consensus.cc:513] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.063097 2221 ts_tablet_manager.cc:1428] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 08:04:30.063153 2278 leader_election.cc:290] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:30.063843 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40af5067b7224accb99733af91d6ca85" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:30.063994 1711 raft_consensus.cc:2466] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:30.064249 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40af5067b7224accb99733af91d6ca85" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:30.064746 1784 leader_election.cc:304] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
W20250625 08:04:30.064973 1784 leader_election.cc:343] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:04:30.065354 2278 raft_consensus.cc:2802] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:30.065411 2278 raft_consensus.cc:491] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:30.065448 2278 raft_consensus.cc:3058] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.066663 2278 raft_consensus.cc:513] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.067019 2278 leader_election.cc:290] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:30.067582 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40af5067b7224accb99733af91d6ca85" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:30.067582 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40af5067b7224accb99733af91d6ca85" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:30.067708 1711 raft_consensus.cc:3058] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
W20250625 08:04:30.067961 1784 leader_election.cc:343] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:04:30.068261 2221 tablet_bootstrap.cc:492] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:30.068732 1711 raft_consensus.cc:2466] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:30.069234 1784 leader_election.cc:304] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130
I20250625 08:04:30.069491 2221 tablet_bootstrap.cc:654] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.069530 2278 raft_consensus.cc:2802] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:30.069598 2278 raft_consensus.cc:695] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:30.069742 2278 consensus_queue.cc:237] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.071246 2221 tablet_bootstrap.cc:492] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:30.071382 2221 ts_tablet_manager.cc:1397] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:30.071851 2221 raft_consensus.cc:357] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.072150 2221 raft_consensus.cc:383] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.072242 2221 raft_consensus.cc:738] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.072417 2221 consensus_queue.cc:260] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.072844 2221 ts_tablet_manager.cc:1428] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:30.073740 2256 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:30.073840 2256 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.074254 2256 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:30.074717 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "579f822528014659a13b9dc93111624f" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:30.074750 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "579f822528014659a13b9dc93111624f" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:30.074841 1847 raft_consensus.cc:2466] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:30.074865 1711 raft_consensus.cc:2466] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:30.075186 1919 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:30.075387 2256 raft_consensus.cc:2802] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:30.075464 2256 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:30.075521 2256 raft_consensus.cc:3058] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.076714 2256 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.076759 1828 tablet_service.cc:1468] Processing CreateTablet for tablet 99a2e60d7c7a4c419b609aa64d8b7334 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:30.077026 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99a2e60d7c7a4c419b609aa64d8b7334. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:30.077126 2256 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:30.079056 2221 tablet_bootstrap.cc:492] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:30.079679 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "579f822528014659a13b9dc93111624f" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:30.079802 1847 raft_consensus.cc:3058] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.080221 2221 tablet_bootstrap.cc:654] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.080789 1847 raft_consensus.cc:2466] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:30.081087 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "579f822528014659a13b9dc93111624f" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:30.081221 1711 raft_consensus.cc:3058] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.081471 2221 tablet_bootstrap.cc:492] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:30.081549 2221 ts_tablet_manager.cc:1397] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:30.082018 2221 raft_consensus.cc:357] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.082120 2221 raft_consensus.cc:383] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.082173 2221 raft_consensus.cc:738] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.082259 1711 raft_consensus.cc:2466] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:30.082325 2221 consensus_queue.cc:260] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.082727 2221 ts_tablet_manager.cc:1428] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:30.083114 1919 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:30.083427 2256 raft_consensus.cc:2802] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:30.083520 2256 raft_consensus.cc:695] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:30.083657 2256 consensus_queue.cc:237] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.087728 2222 tablet_bootstrap.cc:492] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:30.087844 2222 ts_tablet_manager.cc:1397] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.002s
I20250625 08:04:30.087941 2232 ts_tablet_manager.cc:1905] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:30.088387 2222 raft_consensus.cc:357] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.088490 2222 raft_consensus.cc:383] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.088516 2222 raft_consensus.cc:738] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.088639 2222 consensus_queue.cc:260] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.088903 2222 ts_tablet_manager.cc:1428] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:30.089180 2222 tablet_bootstrap.cc:492] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:30.090248 2222 tablet_bootstrap.cc:654] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.090320 2232 ts_tablet_manager.cc:1918] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:30.090389 2232 log.cc:1199] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/2a6db42100cc4b5ca65202e34514e966
I20250625 08:04:30.091264 2232 ts_tablet_manager.cc:1939] T 2a6db42100cc4b5ca65202e34514e966 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:30.092059 2106 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet 2a6db42100cc4b5ca65202e34514e966 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:30.094194 1847 raft_consensus.cc:1273] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Refusing update from remote peer 5dae8595b9484b57bc7ccea543f83130: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 08:04:30.094786 2228 consensus_queue.cc:1035] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.094903 1711 raft_consensus.cc:1273] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Refusing update from remote peer 5dae8595b9484b57bc7ccea543f83130: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 08:04:30.095510 2228 consensus_queue.cc:1035] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.097668 2222 tablet_bootstrap.cc:492] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:30.097757 2222 ts_tablet_manager.cc:1397] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 08:04:30.098321 2222 raft_consensus.cc:357] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.098423 2222 raft_consensus.cc:383] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.098486 2222 raft_consensus.cc:738] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.098590 2222 consensus_queue.cc:260] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.098896 2222 ts_tablet_manager.cc:1428] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:30.099753 2223 tablet_bootstrap.cc:492] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:30.099848 2223 ts_tablet_manager.cc:1397] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20250625 08:04:30.100298 2223 raft_consensus.cc:357] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.100391 2223 raft_consensus.cc:383] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.100418 2223 raft_consensus.cc:738] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.100530 2223 consensus_queue.cc:260] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.100766 2223 ts_tablet_manager.cc:1428] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:30.100944 2223 tablet_bootstrap.cc:492] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:30.101994 2223 tablet_bootstrap.cc:654] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:30.108318 2309 tablet.cc:1722] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 08:04:30.121490 2228 raft_consensus.cc:491] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:30.121625 2228 raft_consensus.cc:513] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.122015 2228 leader_election.cc:290] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:30.122502 2118 catalog_manager.cc:3460] renamed table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020] in HMS: new name default.table_6894ca710ded4040ba337309d454d090
I20250625 08:04:30.122668 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e0682e2bfe43b7a445f595d8929034" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:30.122807 1848 raft_consensus.cc:2466] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:30.122819 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e0682e2bfe43b7a445f595d8929034" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:30.122959 1711 raft_consensus.cc:2466] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:30.124636 2310 tablet.cc:1722] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 08:04:30.124779 2270 raft_consensus.cc:491] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:30.124869 2270 raft_consensus.cc:513] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.125272 2270 leader_election.cc:290] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:30.125802 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a2e60d7c7a4c419b609aa64d8b7334" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
W20250625 08:04:30.126237 1647 leader_election.cc:343] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:04:30.126976 1919 leader_election.cc:304] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:30.127333 2256 raft_consensus.cc:2802] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:30.127393 2256 raft_consensus.cc:491] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:30.127431 2256 raft_consensus.cc:3058] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.128295 2256 raft_consensus.cc:513] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.128619 2256 leader_election.cc:290] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:30.129000 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e0682e2bfe43b7a445f595d8929034" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:30.129093 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a2e60d7c7a4c419b609aa64d8b7334" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:30.129119 1848 raft_consensus.cc:3058] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.129179 1847 raft_consensus.cc:2466] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:30.130633 1848 raft_consensus.cc:2466] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:30.131016 1647 leader_election.cc:304] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130
I20250625 08:04:30.131186 2270 raft_consensus.cc:2802] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:30.131234 2270 raft_consensus.cc:491] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:30.131273 2270 raft_consensus.cc:3058] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.131526 1919 leader_election.cc:304] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:30.132087 2270 raft_consensus.cc:513] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.132421 2270 leader_election.cc:290] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:30.132632 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e0682e2bfe43b7a445f595d8929034" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:30.132726 1711 raft_consensus.cc:3058] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.132782 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a2e60d7c7a4c419b609aa64d8b7334" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
W20250625 08:04:30.133206 1647 leader_election.cc:343] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:04:30.133574 1711 raft_consensus.cc:2466] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:30.133899 2228 raft_consensus.cc:2802] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:30.134037 2228 raft_consensus.cc:695] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:30.134178 2228 consensus_queue.cc:237] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.135751 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a2e60d7c7a4c419b609aa64d8b7334" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:30.135869 1848 raft_consensus.cc:3058] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.136821 1848 raft_consensus.cc:2466] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:30.137238 1647 leader_election.cc:304] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130
F20250625 08:04:30.137259 2182 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 4 out of 11 received events: NotificationEvent(eventId=21, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_31023e55a4384ac88f8415871e52d142, message={"server":"","servicePrincipal":"","db":"default","table":"table_31023e55a4384ac88f8415871e52d142","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_31023e55a4384ac88f8415871e52d142\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_31023e55a4384ac88f8415871e52d142\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_31023e55a4384ac88f8415871e52d142\",\"kudu.table_id\":\"7825ea67af2d42b68200097030b08116\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1750838669, eventType=ALTER_TABLE, dbName=default, tableName=table_7cb71508c2934e1aa091c71056ae7a01, message={"server":"","servicePrincipal":"","db":"default","table":"table_b49cbf7ee55042b39d7b5d3946a35618","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_b49cbf7ee55042b39d7b5d3946a35618\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_b49cbf7ee55042b39d7b5d3946a35618\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b49cbf7ee55042b39d7b5d3946a35618\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7cb71508c2934e1aa091c71056ae7a01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_7cb71508c2934e1aa091c71056ae7a01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7cb71508c2934e1aa091c71056ae7a01\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_d78071777f3646548f7f75cea674ecdc, message={"server":"","servicePrincipal":"","db":"default","table":"table_d78071777f3646548f7f75cea674ecdc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d78071777f3646548f7f75cea674ecdc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_d78071777f3646548f7f75cea674ecdc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d78071777f3646548f7f75cea674ecdc\",\"kudu.table_id\":\"0e43d85f40e545d9b31c5cb3e62f89a8\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=CREATE_TABLE, dbName=default, tableName=table_88079d64160a4318bf8ea1807cb3a30d, message={"server":"","servicePrincipal":"","db":"default","table":"table_88079d64160a4318bf8ea1807cb3a30d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_88079d64160a4318bf8ea1807cb3a30d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_88079d64160a4318bf8ea1807cb3a30d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_88079d64160a4318bf8ea1807cb3a30d\",\"kudu.table_id\":\"7c7e3cbdaf064697a864873ed87a21b6\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250625 08:04:30.138852 2270 raft_consensus.cc:2802] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:30.138964 2270 raft_consensus.cc:695] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:30.139137 2270 consensus_queue.cc:237] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
@ 0x7ffa076f6dcd google::LogMessage::Fail() at ??:0
@ 0x7ffa076fab93 google::LogMessage::SendToLog() at ??:0
@ 0x7ffa076f67cc google::LogMessage::Flush() at ??:0
I20250625 08:04:30.147501 2303 mvcc.cc:204] Tried to move back new op lower bound from 7171435192513576960 to 7171435191836975104. Current Snapshot: MvccSnapshot[applied={T|T < 7171435192513576960}]
@ 0x7ffa076f6a11 google::LogMessage::~LogMessage() at ??:0
I20250625 08:04:30.147894 2314 tablet.cc:1722] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 08:04:30.148350 2223 tablet_bootstrap.cc:492] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:30.148448 2223 ts_tablet_manager.cc:1397] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.001s
I20250625 08:04:30.148900 2223 raft_consensus.cc:357] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.148999 2223 raft_consensus.cc:383] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:30.149026 2223 raft_consensus.cc:738] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:30.149147 2223 consensus_queue.cc:260] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.149397 2223 ts_tablet_manager.cc:1428] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 08:04:30.156953 2278 raft_consensus.cc:491] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:30.157063 2278 raft_consensus.cc:513] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.157402 2278 leader_election.cc:290] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:30.157846 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf12ce8b7b43487aa8bfb6b1e9f88816" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:30.157986 1983 raft_consensus.cc:2466] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:30.158260 1784 leader_election.cc:304] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:30.158641 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf12ce8b7b43487aa8bfb6b1e9f88816" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:30.158759 1711 raft_consensus.cc:2466] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:30.159119 2278 raft_consensus.cc:2802] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:30.159179 2278 raft_consensus.cc:491] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:30.159214 2278 raft_consensus.cc:3058] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
@ 0x7ffa0c68fc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 08:04:30.160034 2278 raft_consensus.cc:513] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.160373 2278 leader_election.cc:290] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:30.160737 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf12ce8b7b43487aa8bfb6b1e9f88816" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:30.160827 1711 raft_consensus.cc:3058] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.161485 1711 raft_consensus.cc:2466] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:30.161710 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf12ce8b7b43487aa8bfb6b1e9f88816" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:30.161782 1784 leader_election.cc:304] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:30.161828 1983 raft_consensus.cc:3058] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.162537 2278 raft_consensus.cc:2802] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:30.162606 1983 raft_consensus.cc:2466] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:30.162622 2278 raft_consensus.cc:695] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:30.162760 2278 consensus_queue.cc:237] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
@ 0x7ffa0c68eba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7ffa0c68e30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7ffa0c693a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x557b3bf1d752 std::function<>::operator()() at ??:0
@ 0x7ffa08b480b7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffa093376db start_thread at ??:0
@ 0x7ffa068ad71f clone at ??:0
I20250625 08:04:30.218086 2233 raft_consensus.cc:491] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:30.218242 2233 raft_consensus.cc:513] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:30.218722 2233 leader_election.cc:290] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:30.219137 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7342ad43df3d493b994075493c6ab8a4" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:30.219218 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7342ad43df3d493b994075493c6ab8a4" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:30.219281 1848 raft_consensus.cc:2466] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:30.219377 1983 raft_consensus.cc:2466] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:30.219589 1647 leader_election.cc:304] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:30.219770 2233 raft_consensus.cc:2802] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:30.219837 2233 raft_consensus.cc:491] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:30.219875 2233 raft_consensus.cc:3058] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.220983 2233 raft_consensus.cc:513] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:30.221402 2233 leader_election.cc:290] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:30.221820 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7342ad43df3d493b994075493c6ab8a4" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:30.221845 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7342ad43df3d493b994075493c6ab8a4" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:30.221951 1848 raft_consensus.cc:3058] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.221956 1983 raft_consensus.cc:3058] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.222786 1983 raft_consensus.cc:2466] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:30.222965 1848 raft_consensus.cc:2466] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:30.223145 1647 leader_election.cc:304] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:30.223340 2233 raft_consensus.cc:2802] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:30.223421 2233 raft_consensus.cc:695] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:30.223565 2233 consensus_queue.cc:237] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
W20250625 08:04:30.341434 2037 connection.cc:537] client connection to 127.1.19.124:37745 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:30.341966 2037 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.19.124:37745)
W20250625 08:04:30.342185 2037 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.124:37745)
W20250625 08:04:30.342543 2037 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.19.124:37745)
W20250625 08:04:30.342923 2037 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.19.124:37745)
I20250625 08:04:30.389822 1101 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu with pid 1345
W20250625 08:04:30.396184 2039 connection.cc:537] client connection to 127.1.19.126:36875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 08:04:30.464066 2278 consensus_queue.cc:1035] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.468248 1983 raft_consensus.cc:3058] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.470001 2334 consensus_queue.cc:1035] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.532562 2308 consensus_queue.cc:1035] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.536345 2228 consensus_queue.cc:1035] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.551033 2341 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:30.551180 2341 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:30.551642 2341 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:30.552974 1434 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:04:30.553069 1432 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:30.553153 1432 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:30.553303 2341 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 08:04:30.573145 2233 consensus_queue.cc:1035] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.577865 2346 consensus_queue.cc:1035] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.594463 2228 consensus_queue.cc:1035] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.598609 2308 consensus_queue.cc:1035] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.611088 2308 raft_consensus.cc:491] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:30.611258 2308 raft_consensus.cc:513] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:30.611738 2308 leader_election.cc:290] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:30.612253 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a2e60d7c7a4c419b609aa64d8b7334" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:30.612242 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a2e60d7c7a4c419b609aa64d8b7334" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:30.612443 1848 raft_consensus.cc:2391] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5dae8595b9484b57bc7ccea543f83130 in current term 1: Already voted for candidate eebfa8472fe04080ac012e2c91bedaae in this term.
I20250625 08:04:30.612779 1919 leader_election.cc:304] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130; no voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:30.612951 2308 raft_consensus.cc:3058] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:30.613827 2308 raft_consensus.cc:2747] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 08:04:30.616580 2346 consensus_queue.cc:1035] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.620370 2346 consensus_queue.cc:1035] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.681995 2334 consensus_queue.cc:1035] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.685271 2278 consensus_queue.cc:1035] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.731739 2346 consensus_queue.cc:1035] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:30.736191 2233 consensus_queue.cc:1035] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:31.250761 2356 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:31.250901 2356 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:31.251381 2356 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:31.251892 1434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250625 08:04:31.252628 1432 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:31.252859 1434 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:04:31.252931 1434 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:31.253206 2356 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 08:04:31.359285 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.126:36875
--webserver_interface=127.1.19.126
--webserver_port=32959
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 08:04:31.404237 1415 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:04:31.404417 1415 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:04:31.474116 2360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:31.474392 2360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:31.474458 2360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:31.477854 2360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:31.477952 2360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:31.477974 2360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:31.477996 2360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:31.482123 2360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.126:36875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.19.126
--webserver_port=32959
--never_fsync=true
--heap_profile_path=/tmp/kudu.2360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:31.483410 2360 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:31.484642 2360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:31.491850 2365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:31.491920 2366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:31.492049 2360 server_base.cc:1048] running on GCE node
W20250625 08:04:31.491850 2368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:31.492811 2360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:31.493865 2360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:31.495141 2360 hybrid_clock.cc:648] HybridClock initialized: now 1750838671495115 us; error 44 us; skew 500 ppm
I20250625 08:04:31.500721 2360 webserver.cc:469] Webserver started at http://127.1.19.126:32959/ using document root <none> and password file <none>
I20250625 08:04:31.501351 2360 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:31.501426 2360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:31.505309 2360 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20250625 08:04:31.507544 2374 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:31.508651 2360 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250625 08:04:31.508772 2360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
format_stamp: "Formatted at 2025-06-25 08:04:25 on dist-test-slave-4qtp"
I20250625 08:04:31.509213 2360 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:31.524544 2360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:31.525295 2360 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:31.525499 2360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:31.533080 2426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.126:36875 every 8 connection(s)
I20250625 08:04:31.533077 2360 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.126:36875
I20250625 08:04:31.534329 2360 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
I20250625 08:04:31.535775 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 2360
I20250625 08:04:31.539018 2427 sys_catalog.cc:263] Verifying existing consensus state
I20250625 08:04:31.541168 2427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap starting.
I20250625 08:04:31.551528 2427 log.cc:826] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:31.580199 2427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=34 ignored=0} mutations{seen=34 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:31.580720 2427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap complete.
I20250625 08:04:31.582666 1895 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:31.584045 2427 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:31.584399 2427 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0b114bd68344b0bd2ee322bd5eccdc, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.584857 2427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 4.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:31.585326 2432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:31.585465 2432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:31.586021 2427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:31.592187 2446 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:31.592370 2446 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: loading cluster ID for follower catalog manager: success
I20250625 08:04:31.595165 2446 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: acquiring CA information for follower catalog manager: success
I20250625 08:04:31.597107 2446 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:31.606191 2360 master_runner.cc:186] Error getting master registration for 127.1.19.124:37745: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111),
I20250625 08:04:31.608898 2360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:31.652041 2030 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:31.653585 2391 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" instance_seqno: 1750838667205184) as {username='slave'} at 127.1.19.67:54387; Asking this server to re-register.
I20250625 08:04:31.654067 2030 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:31.656464 2391 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:31.670290 1758 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:31.671509 2391 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" instance_seqno: 1750838666815429) as {username='slave'} at 127.1.19.65:34767; Asking this server to re-register.
I20250625 08:04:31.671933 1758 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:31.672830 2391 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:31.759423 2456 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:31.759549 2456 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:31.759931 2456 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:31.761286 1432 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:31.763679 2401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 5 candidate_status { last_received { term: 4 index: 53 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
I20250625 08:04:31.764055 2401 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0404561252194ce29bf2924fd5e67e22 in term 4.
I20250625 08:04:31.764406 1434 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 755093bec4b54a9588598f814c3708e1
I20250625 08:04:31.764575 2456 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250625 08:04:31.764639 2456 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:31.764683 2456 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 4 FOLLOWER]: Advancing to term 5
I20250625 08:04:31.765609 2456 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:31.765974 2456 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 election: Requested vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:31.766295 2401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 5 candidate_status { last_received { term: 4 index: 53 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
I20250625 08:04:31.766461 2401 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 4 FOLLOWER]: Advancing to term 5
I20250625 08:04:31.768237 2401 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0404561252194ce29bf2924fd5e67e22 in term 5.
I20250625 08:04:31.768555 1434 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters:
I20250625 08:04:31.768983 2456 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 FOLLOWER]: Leader election won for term 5
I20250625 08:04:31.769219 2456 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 LEADER]: Becoming Leader. State: Replica: 0404561252194ce29bf2924fd5e67e22, State: Running, Role: LEADER
I20250625 08:04:31.769389 2456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 4.53, Last appended by leader: 53, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
W20250625 08:04:31.770129 1432 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:31.770166 2459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0404561252194ce29bf2924fd5e67e22. Latest consensus state: current_term: 5 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:31.770248 2459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:31.770591 2461 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:31.772382 2461 catalog_manager.cc:671] Loaded metadata for table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]
I20250625 08:04:31.772693 2461 catalog_manager.cc:671] Loaded metadata for table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]
I20250625 08:04:31.772953 2461 catalog_manager.cc:671] Loaded metadata for table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]
I20250625 08:04:31.773196 2461 catalog_manager.cc:671] Loaded metadata for table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]
I20250625 08:04:31.773406 2461 catalog_manager.cc:671] Loaded metadata for table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]
I20250625 08:04:31.773603 2461 catalog_manager.cc:671] Loaded metadata for table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]
I20250625 08:04:31.773800 2461 catalog_manager.cc:671] Loaded metadata for table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]
I20250625 08:04:31.774011 2461 catalog_manager.cc:671] Loaded metadata for table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]
I20250625 08:04:31.775475 2461 tablet_loader.cc:96] loaded metadata for tablet 17e0682e2bfe43b7a445f595d8929034 (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6])
I20250625 08:04:31.775888 2461 tablet_loader.cc:96] loaded metadata for tablet 40af5067b7224accb99733af91d6ca85 (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a])
I20250625 08:04:31.776057 2461 tablet_loader.cc:96] loaded metadata for tablet 579f822528014659a13b9dc93111624f (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020])
I20250625 08:04:31.776221 2461 tablet_loader.cc:96] loaded metadata for tablet 626d783b9280477a8dd9ee5cc7f79237 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089])
I20250625 08:04:31.776391 2461 tablet_loader.cc:96] loaded metadata for tablet 7342ad43df3d493b994075493c6ab8a4 (table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904])
I20250625 08:04:31.776548 2461 tablet_loader.cc:96] loaded metadata for tablet 99a2e60d7c7a4c419b609aa64d8b7334 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875])
I20250625 08:04:31.777029 2461 tablet_loader.cc:96] loaded metadata for tablet bf12ce8b7b43487aa8bfb6b1e9f88816 (table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d])
I20250625 08:04:31.777200 2461 tablet_loader.cc:96] loaded metadata for tablet c8549c4ed76541faad94532cd923745c (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206])
I20250625 08:04:31.777619 2461 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:31.778362 2461 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:31.778422 2461 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:31.779357 2461 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:31.779951 2461 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Loaded TSK: 0
I20250625 08:04:31.780268 2461 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:31.781138 2461 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:31.781761 2461 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 20
I20250625 08:04:31.812969 1445 master_service.cc:948] ReplaceTablet: received request to replace tablet c8549c4ed76541faad94532cd923745c from {username='slave'} at 127.0.0.1:35718
I20250625 08:04:31.813061 1438 master_service.cc:948] ReplaceTablet: received request to replace tablet 40af5067b7224accb99733af91d6ca85 from {username='slave'} at 127.0.0.1:35718
I20250625 08:04:31.813398 1447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_174089cfe43b4dbb9ed1b9c2921f9187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:31.813501 1444 master_service.cc:948] ReplaceTablet: received request to replace tablet bf12ce8b7b43487aa8bfb6b1e9f88816 from {username='slave'} at 127.0.0.1:35718
I20250625 08:04:31.813573 1442 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35718:
table { table_name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2" } modify_external_catalogs: true
I20250625 08:04:31.813689 1442 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35718:
table { table_name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2" } modify_external_catalogs: true
I20250625 08:04:31.813704 1440 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_128075ffc1db4314b52b7c28536f6e89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:31.814348 1439 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:31.814570 1446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_bd43623a5f9f40c58ad750234777ba9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:31.814684 1443 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_035cffaa28ec4a838bb218bb37230370"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:31.814760 1441 master_service.cc:948] ReplaceTablet: received request to replace tablet c8549c4ed76541faad94532cd923745c from {username='slave'} at 127.0.0.1:35718
W20250625 08:04:31.817073 1432 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 -> Peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Couldn't send request to peer 755093bec4b54a9588598f814c3708e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:31.817808 2401 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Refusing update from remote peer 0404561252194ce29bf2924fd5e67e22: Log matching property violated. Preceding OpId in replica: term: 4 index: 53. Preceding OpId from leader: term: 5 index: 55. (index mismatch)
F20250625 08:04:31.819972 1612 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 4 out of 11 received events: NotificationEvent(eventId=21, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_31023e55a4384ac88f8415871e52d142, message={"server":"","servicePrincipal":"","db":"default","table":"table_31023e55a4384ac88f8415871e52d142","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_31023e55a4384ac88f8415871e52d142\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_31023e55a4384ac88f8415871e52d142\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_31023e55a4384ac88f8415871e52d142\",\"kudu.table_id\":\"7825ea67af2d42b68200097030b08116\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1750838669, eventType=ALTER_TABLE, dbName=default, tableName=table_7cb71508c2934e1aa091c71056ae7a01, message={"server":"","servicePrincipal":"","db":"default","table":"table_b49cbf7ee55042b39d7b5d3946a35618","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_b49cbf7ee55042b39d7b5d3946a35618\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_b49cbf7ee55042b39d7b5d3946a35618\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b49cbf7ee55042b39d7b5d3946a35618\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7cb71508c2934e1aa091c71056ae7a01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_7cb71508c2934e1aa091c71056ae7a01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7cb71508c2934e1aa091c71056ae7a01\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_d78071777f3646548f7f75cea674ecdc, message={"server":"","servicePrincipal":"","db":"default","table":"table_d78071777f3646548f7f75cea674ecdc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d78071777f3646548f7f75cea674ecdc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_d78071777f3646548f7f75cea674ecdc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d78071777f3646548f7f75cea674ecdc\",\"kudu.table_id\":\"0e43d85f40e545d9b31c5cb3e62f89a8\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=CREATE_TABLE, dbName=default, tableName=table_88079d64160a4318bf8ea1807cb3a30d, message={"server":"","servicePrincipal":"","db":"default","table":"table_88079d64160a4318bf8ea1807cb3a30d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_88079d64160a4318bf8ea1807cb3a30d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_88079d64160a4318bf8ea1807cb3a30d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_88079d64160a4318bf8ea1807cb3a30d\",\"kudu.table_id\":\"7c7e3cbdaf064697a864873ed87a21b6\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250625 08:04:31.821650 2459 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
@ 0x7f217dfbadcd google::LogMessage::Fail() at ??:0
I20250625 08:04:31.826049 2432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0404561252194ce29bf2924fd5e67e22. Latest consensus state: current_term: 5 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:31.826184 2432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f217dfbeb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f217dfba7cc google::LogMessage::Flush() at ??:0
@ 0x7f217dfbaa11 google::LogMessage::~LogMessage() at ??:0
I20250625 08:04:31.831558 2456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:31.831691 2456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:31.832059 2432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:31.832162 2432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:31.833197 2463 mvcc.cc:204] Tried to move back new op lower bound from 7171435199759634432 to 7171435199568863232. Current Snapshot: MvccSnapshot[applied={T|T < 7171435199755993088}]
I20250625 08:04:31.833981 1444 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet bf12ce8b7b43487aa8bfb6b1e9f88816
I20250625 08:04:31.833985 1438 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet 40af5067b7224accb99733af91d6ca85
I20250625 08:04:31.835249 1444 catalog_manager.cc:6652] ReplaceTablet: tablet bf12ce8b7b43487aa8bfb6b1e9f88816 deleted and replaced by tablet 1564b8dab2e9429f9c2a98f0935928f5
I20250625 08:04:31.835251 1445 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet c8549c4ed76541faad94532cd923745c
I20250625 08:04:31.835453 1444 master_service.cc:948] ReplaceTablet: received request to replace tablet 626d783b9280477a8dd9ee5cc7f79237 from {username='slave'} at 127.0.0.1:35718
I20250625 08:04:31.835657 1445 catalog_manager.cc:6652] ReplaceTablet: tablet c8549c4ed76541faad94532cd923745c deleted and replaced by tablet 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:31.835680 1441 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35718:
table { table_name: "default.Table_b49cbf7ee55042b39d7b5d3946a35618" } new_table_name: "default.Table_7cb71508c2934e1aa091c71056ae7a01" modify_external_catalogs: true
I20250625 08:04:31.835860 1445 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35718:
name: "default.Table_31023e55a4384ac88f8415871e52d142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f2182f53c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 08:04:31.838833 1438 catalog_manager.cc:6652] ReplaceTablet: tablet 40af5067b7224accb99733af91d6ca85 deleted and replaced by tablet bcdaf64470204ff4986224f728a51ac8
I20250625 08:04:31.839054 1438 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35718:
table { table_name: "default.Table_5eebaa7c05694491963fce48525e4681" } new_table_name: "default.Table_69a1c4361ab540a48fd24a5198251c2c" modify_external_catalogs: true
@ 0x7f2182f52ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 08:04:31.849742 1444 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet 626d783b9280477a8dd9ee5cc7f79237
I20250625 08:04:31.849995 1444 catalog_manager.cc:6652] ReplaceTablet: tablet 626d783b9280477a8dd9ee5cc7f79237 deleted and replaced by tablet ddb06adb0b7d4f888e489eb5721321fc
I20250625 08:04:31.850188 1444 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35718:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:31.850312 1444 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35718:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:31.853438 1691 tablet_service.cc:1515] Processing DeleteTablet for tablet 40af5067b7224accb99733af91d6ca85 with delete_type TABLET_DATA_DELETED (replaced by tablet bcdaf64470204ff4986224f728a51ac8) from {username='slave'} at 127.0.0.1:32780
I20250625 08:04:31.853482 1690 tablet_service.cc:1515] Processing DeleteTablet for tablet c8549c4ed76541faad94532cd923745c with delete_type TABLET_DATA_DELETED (replaced by tablet 18ea19fdd2814cc08c6315b3463bd3a9) from {username='slave'} at 127.0.0.1:32780
I20250625 08:04:31.853539 1688 tablet_service.cc:1468] Processing CreateTablet for tablet 18ea19fdd2814cc08c6315b3463bd3a9 (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.853719 1689 tablet_service.cc:1515] Processing DeleteTablet for tablet bf12ce8b7b43487aa8bfb6b1e9f88816 with delete_type TABLET_DATA_DELETED (replaced by tablet 1564b8dab2e9429f9c2a98f0935928f5) from {username='slave'} at 127.0.0.1:32780
I20250625 08:04:31.853765 1687 tablet_service.cc:1468] Processing CreateTablet for tablet bcdaf64470204ff4986224f728a51ac8 (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.854185 1686 tablet_service.cc:1468] Processing CreateTablet for tablet 1564b8dab2e9429f9c2a98f0935928f5 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.854488 1685 tablet_service.cc:1515] Processing DeleteTablet for tablet 626d783b9280477a8dd9ee5cc7f79237 with delete_type TABLET_DATA_DELETED (replaced by tablet ddb06adb0b7d4f888e489eb5721321fc) from {username='slave'} at 127.0.0.1:32780
I20250625 08:04:31.853814 1688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ea19fdd2814cc08c6315b3463bd3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.855686 1685 tablet_service.cc:1468] Processing CreateTablet for tablet ddb06adb0b7d4f888e489eb5721321fc (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.855930 1685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb06adb0b7d4f888e489eb5721321fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.857333 1687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcdaf64470204ff4986224f728a51ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.857460 2481 tablet_replica.cc:331] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:31.857589 2481 raft_consensus.cc:2241] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:31.857695 2481 raft_consensus.cc:2270] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.858103 2481 ts_tablet_manager.cc:1905] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.859318 1686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1564b8dab2e9429f9c2a98f0935928f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.860498 2481 ts_tablet_manager.cc:1918] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:31.860558 2481 log.cc:1199] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/40af5067b7224accb99733af91d6ca85
I20250625 08:04:31.860836 2481 ts_tablet_manager.cc:1939] T 40af5067b7224accb99733af91d6ca85 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:31.861429 2481 tablet_replica.cc:331] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:31.861546 2481 raft_consensus.cc:2241] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:31.861524 1434 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 40af5067b7224accb99733af91d6ca85 (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]) successfully deleted
I20250625 08:04:31.861768 2481 raft_consensus.cc:2270] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.861753 1828 tablet_service.cc:1515] Processing DeleteTablet for tablet bf12ce8b7b43487aa8bfb6b1e9f88816 with delete_type TABLET_DATA_DELETED (replaced by tablet 1564b8dab2e9429f9c2a98f0935928f5) from {username='slave'} at 127.0.0.1:56098
I20250625 08:04:31.862000 1826 tablet_service.cc:1515] Processing DeleteTablet for tablet c8549c4ed76541faad94532cd923745c with delete_type TABLET_DATA_DELETED (replaced by tablet 18ea19fdd2814cc08c6315b3463bd3a9) from {username='slave'} at 127.0.0.1:56098
I20250625 08:04:31.862100 1827 tablet_service.cc:1515] Processing DeleteTablet for tablet 40af5067b7224accb99733af91d6ca85 with delete_type TABLET_DATA_DELETED (replaced by tablet bcdaf64470204ff4986224f728a51ac8) from {username='slave'} at 127.0.0.1:56098
I20250625 08:04:31.862228 2481 ts_tablet_manager.cc:1905] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.862399 1825 tablet_service.cc:1468] Processing CreateTablet for tablet 18ea19fdd2814cc08c6315b3463bd3a9 (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.862687 1825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ea19fdd2814cc08c6315b3463bd3a9. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f2182f5230b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 08:04:31.864619 2481 ts_tablet_manager.cc:1918] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:31.864691 2481 log.cc:1199] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/c8549c4ed76541faad94532cd923745c
I20250625 08:04:31.864972 2481 ts_tablet_manager.cc:1939] T c8549c4ed76541faad94532cd923745c P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:31.865165 1824 tablet_service.cc:1468] Processing CreateTablet for tablet bcdaf64470204ff4986224f728a51ac8 (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.865420 1824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcdaf64470204ff4986224f728a51ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.865648 2481 tablet_replica.cc:331] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:31.865761 2481 raft_consensus.cc:2241] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:31.865768 1434 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet c8549c4ed76541faad94532cd923745c (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]) successfully deleted
I20250625 08:04:31.865865 2481 raft_consensus.cc:2270] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.866070 2482 tablet_bootstrap.cc:492] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:31.866245 2481 ts_tablet_manager.cc:1905] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.867156 2482 tablet_bootstrap.cc:654] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.867681 1823 tablet_service.cc:1468] Processing CreateTablet for tablet 1564b8dab2e9429f9c2a98f0935928f5 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.867934 1823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1564b8dab2e9429f9c2a98f0935928f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.868348 2481 ts_tablet_manager.cc:1918] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:31.868420 2481 log.cc:1199] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/bf12ce8b7b43487aa8bfb6b1e9f88816
@ 0x7f2182f57a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250625 08:04:31.868695 2481 ts_tablet_manager.cc:1939] T bf12ce8b7b43487aa8bfb6b1e9f88816 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
@ 0x55a0ae3dd752 std::function<>::operator()() at ??:0
I20250625 08:04:31.869621 1434 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet bf12ce8b7b43487aa8bfb6b1e9f88816 (table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]) successfully deleted
I20250625 08:04:31.869968 1822 tablet_service.cc:1515] Processing DeleteTablet for tablet 626d783b9280477a8dd9ee5cc7f79237 with delete_type TABLET_DATA_DELETED (replaced by tablet ddb06adb0b7d4f888e489eb5721321fc) from {username='slave'} at 127.0.0.1:56098
I20250625 08:04:31.870271 1821 tablet_service.cc:1468] Processing CreateTablet for tablet ddb06adb0b7d4f888e489eb5721321fc (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.870564 1821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb06adb0b7d4f888e489eb5721321fc. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f217f40c0b7 kudu::Thread::SuperviseThread() at ??:0
I20250625 08:04:31.872951 2484 tablet_replica.cc:331] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:31.873085 2484 raft_consensus.cc:2241] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:31.873250 2481 tablet_replica.cc:331] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:31.873335 2484 raft_consensus.cc:2270] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.873414 2481 raft_consensus.cc:2241] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:31.873600 2481 raft_consensus.cc:2270] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.873730 2484 ts_tablet_manager.cc:1905] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.874054 2481 ts_tablet_manager.cc:1905] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.874640 2485 tablet_bootstrap.cc:492] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:31.875778 2485 tablet_bootstrap.cc:654] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.875867 2484 ts_tablet_manager.cc:1918] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:31.875929 2484 log.cc:1199] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/bf12ce8b7b43487aa8bfb6b1e9f88816
I20250625 08:04:31.876206 2484 ts_tablet_manager.cc:1939] T bf12ce8b7b43487aa8bfb6b1e9f88816 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:31.876511 2481 ts_tablet_manager.cc:1918] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 08:04:31.876580 2481 log.cc:1199] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/626d783b9280477a8dd9ee5cc7f79237
I20250625 08:04:31.876817 2481 ts_tablet_manager.cc:1939] T 626d783b9280477a8dd9ee5cc7f79237 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:31.876996 1434 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet bf12ce8b7b43487aa8bfb6b1e9f88816 (table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]) successfully deleted
I20250625 08:04:31.877097 2482 tablet_bootstrap.cc:492] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:31.877172 2482 ts_tablet_manager.cc:1397] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250625 08:04:31.877493 1434 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 626d783b9280477a8dd9ee5cc7f79237 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:31.877717 2482 raft_consensus.cc:357] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.877852 2482 raft_consensus.cc:383] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.877894 2482 raft_consensus.cc:738] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.878106 2482 consensus_queue.cc:260] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.878422 1963 tablet_service.cc:1515] Processing DeleteTablet for tablet bf12ce8b7b43487aa8bfb6b1e9f88816 with delete_type TABLET_DATA_DELETED (replaced by tablet 1564b8dab2e9429f9c2a98f0935928f5) from {username='slave'} at 127.0.0.1:35470
I20250625 08:04:31.878712 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet 40af5067b7224accb99733af91d6ca85 with delete_type TABLET_DATA_DELETED (replaced by tablet bcdaf64470204ff4986224f728a51ac8) from {username='slave'} at 127.0.0.1:35470
I20250625 08:04:31.878825 1961 tablet_service.cc:1515] Processing DeleteTablet for tablet c8549c4ed76541faad94532cd923745c with delete_type TABLET_DATA_DELETED (replaced by tablet 18ea19fdd2814cc08c6315b3463bd3a9) from {username='slave'} at 127.0.0.1:35470
I20250625 08:04:31.879154 1960 tablet_service.cc:1468] Processing CreateTablet for tablet 18ea19fdd2814cc08c6315b3463bd3a9 (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.879645 1959 tablet_service.cc:1468] Processing CreateTablet for tablet bcdaf64470204ff4986224f728a51ac8 (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.879997 1958 tablet_service.cc:1468] Processing CreateTablet for tablet 1564b8dab2e9429f9c2a98f0935928f5 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.880631 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ea19fdd2814cc08c6315b3463bd3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.881642 2482 ts_tablet_manager.cc:1428] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 08:04:31.883227 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1564b8dab2e9429f9c2a98f0935928f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.883359 2482 tablet_bootstrap.cc:492] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:31.885002 2482 tablet_bootstrap.cc:654] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.885414 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcdaf64470204ff4986224f728a51ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.885608 2490 tablet_bootstrap.cc:492] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:31.886749 2490 tablet_bootstrap.cc:654] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.887404 2485 tablet_bootstrap.cc:492] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:31.887508 2485 ts_tablet_manager.cc:1397] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 08:04:31.888254 1959 tablet_service.cc:1515] Processing DeleteTablet for tablet 626d783b9280477a8dd9ee5cc7f79237 with delete_type TABLET_DATA_DELETED (replaced by tablet ddb06adb0b7d4f888e489eb5721321fc) from {username='slave'} at 127.0.0.1:35470
I20250625 08:04:31.888192 2485 raft_consensus.cc:357] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.888340 2485 raft_consensus.cc:383] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.888383 2485 raft_consensus.cc:738] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.888578 2485 consensus_queue.cc:260] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.888593 1958 tablet_service.cc:1468] Processing CreateTablet for tablet ddb06adb0b7d4f888e489eb5721321fc (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:31.888852 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb06adb0b7d4f888e489eb5721321fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:31.890107 2485 ts_tablet_manager.cc:1428] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:31.890329 2485 tablet_bootstrap.cc:492] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:31.890285 2391 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" instance_seqno: 1750838667017582) as {username='slave'} at 127.1.19.66:54569; Asking this server to re-register.
I20250625 08:04:31.890833 1895 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:31.891954 2391 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:31.892310 2485 tablet_bootstrap.cc:654] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.893625 2490 tablet_bootstrap.cc:492] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
@ 0x7f217fbfb6db start_thread at ??:0
I20250625 08:04:31.893784 2490 ts_tablet_manager.cc:1397] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250625 08:04:31.894567 2490 raft_consensus.cc:357] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.894701 2490 raft_consensus.cc:383] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.894740 2490 raft_consensus.cc:738] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.894928 2490 consensus_queue.cc:260] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.895181 2488 tablet_replica.cc:331] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:31.895327 2488 raft_consensus.cc:2241] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:31.895426 2488 raft_consensus.cc:2270] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.895794 2488 ts_tablet_manager.cc:1905] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f217d17171f clone at ??:0
I20250625 08:04:31.896607 2484 tablet_replica.cc:331] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:31.896734 2484 raft_consensus.cc:2241] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:31.896842 2484 raft_consensus.cc:2270] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.897213 2484 ts_tablet_manager.cc:1905] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.897260 2482 tablet_bootstrap.cc:492] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:31.897362 2482 ts_tablet_manager.cc:1397] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20250625 08:04:31.897898 2482 raft_consensus.cc:357] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.898027 2482 raft_consensus.cc:383] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.898062 2482 raft_consensus.cc:738] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.898088 2490 ts_tablet_manager.cc:1428] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 08:04:31.898232 2482 consensus_queue.cc:260] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.899269 2482 ts_tablet_manager.cc:1428] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 08:04:31.899585 2482 tablet_bootstrap.cc:492] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:31.900012 2490 tablet_bootstrap.cc:492] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:31.901083 2490 tablet_bootstrap.cc:654] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.901166 2482 tablet_bootstrap.cc:654] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.902467 2485 tablet_bootstrap.cc:492] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:31.902557 2485 ts_tablet_manager.cc:1397] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20250625 08:04:31.902778 2482 tablet_bootstrap.cc:492] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:31.902849 2482 ts_tablet_manager.cc:1397] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250625 08:04:31.903096 2485 raft_consensus.cc:357] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.903196 2485 raft_consensus.cc:383] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.903242 2485 raft_consensus.cc:738] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.903378 2482 raft_consensus.cc:357] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.903394 2485 consensus_queue.cc:260] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.903466 2482 raft_consensus.cc:383] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.903499 2482 raft_consensus.cc:738] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.903625 2482 consensus_queue.cc:260] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.903676 2485 ts_tablet_manager.cc:1428] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:31.903894 2485 tablet_bootstrap.cc:492] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:31.905102 2485 tablet_bootstrap.cc:654] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.905652 2482 ts_tablet_manager.cc:1428] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 08:04:31.906426 2485 tablet_bootstrap.cc:492] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:31.906554 2485 ts_tablet_manager.cc:1397] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:31.906844 2482 tablet_bootstrap.cc:492] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:31.907342 2485 raft_consensus.cc:357] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.907447 2485 raft_consensus.cc:383] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.907483 2485 raft_consensus.cc:738] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.907630 2485 consensus_queue.cc:260] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.908282 2484 ts_tablet_manager.cc:1918] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:31.908360 2484 log.cc:1199] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/c8549c4ed76541faad94532cd923745c
I20250625 08:04:31.908613 2484 ts_tablet_manager.cc:1939] T c8549c4ed76541faad94532cd923745c P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:31.908687 2482 tablet_bootstrap.cc:654] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.909425 2485 ts_tablet_manager.cc:1428] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250625 08:04:31.909462 2488 ts_tablet_manager.cc:1918] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:31.909538 2488 log.cc:1199] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/bf12ce8b7b43487aa8bfb6b1e9f88816
I20250625 08:04:31.909825 2488 ts_tablet_manager.cc:1939] T bf12ce8b7b43487aa8bfb6b1e9f88816 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:31.910560 2488 tablet_replica.cc:331] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:31.910686 2490 tablet_bootstrap.cc:492] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:31.910780 2490 ts_tablet_manager.cc:1397] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250625 08:04:31.910988 2485 tablet_bootstrap.cc:492] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:31.911266 2490 raft_consensus.cc:357] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.911381 2490 raft_consensus.cc:383] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.911409 2490 raft_consensus.cc:738] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.911556 2490 consensus_queue.cc:260] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.912309 2485 tablet_bootstrap.cc:654] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.913038 2490 ts_tablet_manager.cc:1428] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 08:04:31.913913 2490 tablet_bootstrap.cc:492] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:31.914666 2485 tablet_bootstrap.cc:492] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:31.914667 2482 tablet_bootstrap.cc:492] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:31.914752 2482 ts_tablet_manager.cc:1397] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 08:04:31.914765 2485 ts_tablet_manager.cc:1397] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20250625 08:04:31.915077 2490 tablet_bootstrap.cc:654] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.915234 2485 raft_consensus.cc:357] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.915234 2482 raft_consensus.cc:357] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.915340 2482 raft_consensus.cc:383] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.915340 2485 raft_consensus.cc:383] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.915374 2485 raft_consensus.cc:738] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.915374 2482 raft_consensus.cc:738] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.915517 2485 consensus_queue.cc:260] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.915517 2482 consensus_queue.cc:260] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.915802 2485 ts_tablet_manager.cc:1428] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:31.910681 2488 raft_consensus.cc:2241] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:31.917312 2488 raft_consensus.cc:2270] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.917310 2482 ts_tablet_manager.cc:1428] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 08:04:31.915818 2484 tablet_replica.cc:331] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:31.918222 2484 raft_consensus.cc:2241] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:31.918967 2484 raft_consensus.cc:2270] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.919124 2488 ts_tablet_manager.cc:1905] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.919307 2484 ts_tablet_manager.cc:1905] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.921489 2484 ts_tablet_manager.cc:1918] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:31.921564 2484 log.cc:1199] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/40af5067b7224accb99733af91d6ca85
I20250625 08:04:31.921840 2484 ts_tablet_manager.cc:1939] T 40af5067b7224accb99733af91d6ca85 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:31.922542 2484 tablet_replica.cc:331] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:31.922683 2484 raft_consensus.cc:2241] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:31.922850 2484 raft_consensus.cc:2270] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.923273 2484 ts_tablet_manager.cc:1905] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.925660 2484 ts_tablet_manager.cc:1918] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 08:04:31.925735 2484 log.cc:1199] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/626d783b9280477a8dd9ee5cc7f79237
I20250625 08:04:31.925988 2484 ts_tablet_manager.cc:1939] T 626d783b9280477a8dd9ee5cc7f79237 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:31.926083 2488 ts_tablet_manager.cc:1918] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:31.926192 2488 log.cc:1199] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/40af5067b7224accb99733af91d6ca85
I20250625 08:04:31.926968 2488 ts_tablet_manager.cc:1939] T 40af5067b7224accb99733af91d6ca85 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:31.927919 2490 tablet_bootstrap.cc:492] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:31.928007 2490 ts_tablet_manager.cc:1397] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20250625 08:04:31.928563 2490 raft_consensus.cc:357] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.928673 2490 raft_consensus.cc:383] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.928701 2490 raft_consensus.cc:738] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.928869 2490 consensus_queue.cc:260] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.931429 2228 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:31.931514 2228 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.931869 2228 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:31.932678 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ea19fdd2814cc08c6315b3463bd3a9" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:31.932710 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ea19fdd2814cc08c6315b3463bd3a9" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:31.932821 1711 raft_consensus.cc:2466] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:31.932859 1848 raft_consensus.cc:2466] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:31.933234 1919 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:31.933444 2228 raft_consensus.cc:2802] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:31.933503 2228 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:31.933550 2228 raft_consensus.cc:3058] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:31.933820 2490 ts_tablet_manager.cc:1428] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250625 08:04:31.934840 2228 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.935276 2228 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:31.936046 2490 tablet_bootstrap.cc:492] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:31.936286 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ea19fdd2814cc08c6315b3463bd3a9" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:31.936408 1848 raft_consensus.cc:3058] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:31.936363 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ea19fdd2814cc08c6315b3463bd3a9" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:31.936496 1711 raft_consensus.cc:3058] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:31.937235 2490 tablet_bootstrap.cc:654] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:31.937458 1848 raft_consensus.cc:2466] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:31.937511 1711 raft_consensus.cc:2466] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:31.938004 2488 tablet_replica.cc:331] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:31.937978 1919 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:31.938201 2488 raft_consensus.cc:2241] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:31.938371 2228 raft_consensus.cc:2802] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:31.938376 2488 raft_consensus.cc:2270] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.938532 2490 tablet_bootstrap.cc:492] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:31.938570 2228 raft_consensus.cc:695] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:31.938616 2490 ts_tablet_manager.cc:1397] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20250625 08:04:31.938709 2228 consensus_queue.cc:237] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.938875 2488 ts_tablet_manager.cc:1905] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.939175 2490 raft_consensus.cc:357] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.939368 2490 raft_consensus.cc:383] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:31.939451 2490 raft_consensus.cc:738] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:31.939610 2490 consensus_queue.cc:260] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:31.939982 2490 ts_tablet_manager.cc:1428] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:31.940837 2488 ts_tablet_manager.cc:1918] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:31.940903 2488 log.cc:1199] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/c8549c4ed76541faad94532cd923745c
I20250625 08:04:31.941155 2488 ts_tablet_manager.cc:1939] T c8549c4ed76541faad94532cd923745c P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:31.941761 2488 tablet_replica.cc:331] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:31.941913 2488 raft_consensus.cc:2241] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:31.942124 2488 raft_consensus.cc:2270] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:31.942528 2488 ts_tablet_manager.cc:1905] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:31.944134 2488 ts_tablet_manager.cc:1918] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 08:04:31.944197 2488 log.cc:1199] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/626d783b9280477a8dd9ee5cc7f79237
I20250625 08:04:31.944396 2488 ts_tablet_manager.cc:1939] T 626d783b9280477a8dd9ee5cc7f79237 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:32.083969 2507 raft_consensus.cc:491] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:32.084122 2507 raft_consensus.cc:513] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:32.084614 2507 leader_election.cc:290] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:32.085104 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1564b8dab2e9429f9c2a98f0935928f5" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:32.085172 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1564b8dab2e9429f9c2a98f0935928f5" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:32.085286 1711 raft_consensus.cc:2466] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:32.085330 1848 raft_consensus.cc:2466] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:32.085657 1919 leader_election.cc:304] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:32.085819 2507 raft_consensus.cc:2802] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:32.085878 2507 raft_consensus.cc:491] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:32.085956 2507 raft_consensus.cc:3058] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
W20250625 08:04:32.087174 1781 connection.cc:537] server connection from 127.0.0.1:56098 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:32.087215 2036 connection.cc:537] client connection to 127.1.19.125:38317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 08:04:32.087249 2507 raft_consensus.cc:513] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:32.087627 2507 leader_election.cc:290] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:04:32.087666 2036 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.19.125:38317)
I20250625 08:04:32.088017 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1564b8dab2e9429f9c2a98f0935928f5" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:32.088055 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1564b8dab2e9429f9c2a98f0935928f5" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
W20250625 08:04:32.088145 2036 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.125:38317)
I20250625 08:04:32.088133 1848 raft_consensus.cc:3058] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:32.088196 1711 raft_consensus.cc:3058] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
W20250625 08:04:32.088310 2036 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.19.125:38317)
W20250625 08:04:32.088569 2036 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.19.125:38317)
I20250625 08:04:32.089195 1848 raft_consensus.cc:2466] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:32.089191 1711 raft_consensus.cc:2466] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:32.089465 1919 leader_election.cc:304] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:32.089643 2507 raft_consensus.cc:2802] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:32.089735 2507 raft_consensus.cc:695] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:32.089895 2507 consensus_queue.cc:237] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
W20250625 08:04:32.091187 1916 connection.cc:537] client connection to 127.1.19.125:38317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:32.092844 2331 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:32.089518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.125:38317 (local address 127.0.0.1:49796)
0625 08:04:32.089703 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:32.089720 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:32.089818 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:32.092080 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":47}
W20250625 08:04:32.093044 2455 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:32.091728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.125:38317 (local address 127.1.19.65:47483)
0625 08:04:32.092501 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:32.092552 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":689}
W20250625 08:04:32.094372 1918 connection.cc:537] server connection from 127.0.0.1:35470 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 08:04:32.155329 2498 raft_consensus.cc:491] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:32.155479 2498 raft_consensus.cc:513] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:32.155925 2498 leader_election.cc:290] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:32.156430 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddb06adb0b7d4f888e489eb5721321fc" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:32.156461 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddb06adb0b7d4f888e489eb5721321fc" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:32.156591 1711 raft_consensus.cc:2466] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:32.156621 1983 raft_consensus.cc:2466] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:32.156939 1784 leader_election.cc:304] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:32.157186 2498 raft_consensus.cc:2802] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:32.157262 2498 raft_consensus.cc:491] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:32.157308 2498 raft_consensus.cc:3058] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:32.158317 2346 raft_consensus.cc:491] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:32.158319 2498 raft_consensus.cc:513] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:32.158413 2346 raft_consensus.cc:513] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:32.158701 2498 leader_election.cc:290] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:32.158861 2346 leader_election.cc:290] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:32.159269 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddb06adb0b7d4f888e489eb5721321fc" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:32.159283 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcdaf64470204ff4986224f728a51ac8" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:32.159394 1711 raft_consensus.cc:3058] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:32.159437 1983 raft_consensus.cc:2466] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:32.159283 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddb06adb0b7d4f888e489eb5721321fc" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:32.159626 1982 raft_consensus.cc:3058] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:32.159760 1647 leader_election.cc:304] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:32.159929 2346 raft_consensus.cc:2802] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:32.159983 2346 raft_consensus.cc:491] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:32.160027 2346 raft_consensus.cc:3058] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:32.160486 1982 raft_consensus.cc:2466] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:32.160490 1711 raft_consensus.cc:2466] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:32.160741 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcdaf64470204ff4986224f728a51ac8" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:32.160780 1784 leader_election.cc:304] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:32.160869 1848 raft_consensus.cc:2466] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:32.161022 2346 raft_consensus.cc:513] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:32.161063 2498 raft_consensus.cc:2802] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:32.161157 2498 raft_consensus.cc:695] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:32.161283 2498 consensus_queue.cc:237] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:32.161403 2346 leader_election.cc:290] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:32.162132 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcdaf64470204ff4986224f728a51ac8" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:32.162150 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcdaf64470204ff4986224f728a51ac8" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:32.162254 1982 raft_consensus.cc:3058] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:32.162478 1848 raft_consensus.cc:3058] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:32.163271 1982 raft_consensus.cc:2466] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:32.163435 1848 raft_consensus.cc:2466] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:32.163628 1647 leader_election.cc:304] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:32.163795 2346 raft_consensus.cc:2802] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:32.164021 2346 raft_consensus.cc:695] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:32.164172 2346 consensus_queue.cc:237] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:32.356247 2515 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:32.356391 2515 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:32.357677 2515 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:32.358323 2376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:32.358843 2507 consensus_queue.cc:1035] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 08:04:32.359213 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:32.359416 2376 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:32.359496 2376 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:32.359692 2515 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250625 08:04:32.363444 2228 consensus_queue.cc:1035] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:32.587668 2507 consensus_queue.cc:1035] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:32.592506 2526 consensus_queue.cc:1035] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:32.728783 2346 consensus_queue.cc:1035] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:32.733031 2530 consensus_queue.cc:1035] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:32.746464 2498 consensus_queue.cc:1035] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:32.750836 2533 consensus_queue.cc:1035] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:33.028406 2535 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:33.028549 2535 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:33.029428 2535 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:33.030622 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:33.030722 2376 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:33.030807 2376 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:33.030941 2535 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250625 08:04:34.134233 2554 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:34.134400 2554 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:34.134923 2554 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:34.136164 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:34.136270 2376 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:34.136332 2376 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:34.136472 2554 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250625 08:04:34.830024 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.125:38317
--webserver_interface=127.1.19.125
--webserver_port=39497
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 08:04:34.941913 2569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:34.942236 2569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:34.942308 2569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:34.945744 2569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:34.945823 2569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:34.945858 2569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:34.945890 2569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:34.949951 2569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.125:38317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.19.125
--webserver_port=39497
--never_fsync=true
--heap_profile_path=/tmp/kudu.2569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:34.951226 2569 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:34.952423 2569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:34.959334 2574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:34.959352 2577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:34.959597 2569 server_base.cc:1048] running on GCE node
W20250625 08:04:34.959367 2575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:34.960242 2569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:34.961267 2569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:34.962472 2569 hybrid_clock.cc:648] HybridClock initialized: now 1750838674962450 us; error 38 us; skew 500 ppm
I20250625 08:04:34.964501 2569 webserver.cc:469] Webserver started at http://127.1.19.125:39497/ using document root <none> and password file <none>
I20250625 08:04:34.965140 2569 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:34.965219 2569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:34.968806 2569 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250625 08:04:34.970911 2583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:34.972036 2569 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:34.972177 2569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
uuid: "0404561252194ce29bf2924fd5e67e22"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:34.972610 2569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:35.004247 2569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:35.004958 2569 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:35.005155 2569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:35.012897 2569 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.125:38317
I20250625 08:04:35.012904 2637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.125:38317 every 8 connection(s)
I20250625 08:04:35.014103 2569 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
I20250625 08:04:35.016490 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 2569
I20250625 08:04:35.018397 2638 sys_catalog.cc:263] Verifying existing consensus state
I20250625 08:04:35.020205 2638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap starting.
I20250625 08:04:35.030148 2638 log.cc:826] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:35.062064 2638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=1 applied=60 ignored=0} inserts{seen=38 ignored=0} mutations{seen=42 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:35.062593 2638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap complete.
I20250625 08:04:35.065683 2638 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:35.066051 2638 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0404561252194ce29bf2924fd5e67e22, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.066588 2638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 5.60, Last appended by leader: 60, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:35.067068 2642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:35.067214 2642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:35.067727 2638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:35.076634 2656 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:35.076715 2656 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: loading cluster ID for follower catalog manager: success
I20250625 08:04:35.083988 2656 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: acquiring CA information for follower catalog manager: success
I20250625 08:04:35.086257 2656 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:35.085939 2569 master_runner.cc:186] Error getting master registration for 127.1.19.124:37745: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111),
I20250625 08:04:35.088554 2569 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:35.224537 1896 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:35.226915 2602 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" instance_seqno: 1750838667017582) as {username='slave'} at 127.1.19.66:50809; Asking this server to re-register.
I20250625 08:04:35.227629 1896 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:35.228978 2602 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:35.305651 1759 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:35.306843 2602 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" instance_seqno: 1750838666815429) as {username='slave'} at 127.1.19.65:42639; Asking this server to re-register.
I20250625 08:04:35.307263 1759 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:35.308215 2602 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:35.490401 2031 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:35.491623 2602 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" instance_seqno: 1750838667205184) as {username='slave'} at 127.1.19.67:35415; Asking this server to re-register.
I20250625 08:04:35.492055 2031 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:35.492942 2602 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:35.643658 2668 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:35.643870 2668 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:35.645089 2668 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:35.645601 2401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 6 candidate_status { last_received { term: 5 index: 60 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
W20250625 08:04:35.645747 2585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:35.645764 2401 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0404561252194ce29bf2924fd5e67e22 in term 5.
I20250625 08:04:35.646126 2587 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters:
I20250625 08:04:35.646377 2668 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250625 08:04:35.646478 2668 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:35.646541 2668 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 5 FOLLOWER]: Advancing to term 6
W20250625 08:04:35.646555 2585 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:35.648352 2668 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:35.648684 2668 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 6 election: Requested vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:35.649056 2401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 6 candidate_status { last_received { term: 5 index: 60 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
I20250625 08:04:35.649196 2401 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 5 FOLLOWER]: Advancing to term 6
W20250625 08:04:35.649649 2585 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:35.650202 2401 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0404561252194ce29bf2924fd5e67e22 in term 6.
I20250625 08:04:35.650581 2587 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 755093bec4b54a9588598f814c3708e1
I20250625 08:04:35.650769 2668 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 FOLLOWER]: Leader election won for term 6
I20250625 08:04:35.650925 2668 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 LEADER]: Becoming Leader. State: Replica: 0404561252194ce29bf2924fd5e67e22, State: Running, Role: LEADER
I20250625 08:04:35.651218 2668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 5.60, Last appended by leader: 60, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:35.652896 2671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0404561252194ce29bf2924fd5e67e22. Latest consensus state: current_term: 6 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:35.653013 2671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:35.653321 2673 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:35.654966 2673 catalog_manager.cc:671] Loaded metadata for table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]
I20250625 08:04:35.655283 2673 catalog_manager.cc:671] Loaded metadata for table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]
I20250625 08:04:35.655512 2673 catalog_manager.cc:671] Loaded metadata for table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]
I20250625 08:04:35.655720 2673 catalog_manager.cc:671] Loaded metadata for table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]
I20250625 08:04:35.655918 2673 catalog_manager.cc:671] Loaded metadata for table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]
I20250625 08:04:35.656113 2673 catalog_manager.cc:671] Loaded metadata for table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]
I20250625 08:04:35.656351 2673 catalog_manager.cc:671] Loaded metadata for table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]
I20250625 08:04:35.656569 2673 catalog_manager.cc:671] Loaded metadata for table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]
I20250625 08:04:35.657809 2673 tablet_loader.cc:96] loaded metadata for tablet 1564b8dab2e9429f9c2a98f0935928f5 (table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d])
I20250625 08:04:35.658016 2673 tablet_loader.cc:96] loaded metadata for tablet 17e0682e2bfe43b7a445f595d8929034 (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6])
I20250625 08:04:35.658202 2673 tablet_loader.cc:96] loaded metadata for tablet 18ea19fdd2814cc08c6315b3463bd3a9 (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206])
I20250625 08:04:35.658667 2673 tablet_loader.cc:96] loaded metadata for tablet 579f822528014659a13b9dc93111624f (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020])
I20250625 08:04:35.658917 2673 tablet_loader.cc:96] loaded metadata for tablet 7342ad43df3d493b994075493c6ab8a4 (table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904])
I20250625 08:04:35.659075 2673 tablet_loader.cc:96] loaded metadata for tablet 99a2e60d7c7a4c419b609aa64d8b7334 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875])
I20250625 08:04:35.659512 2673 tablet_loader.cc:96] loaded metadata for tablet bcdaf64470204ff4986224f728a51ac8 (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a])
I20250625 08:04:35.660017 2673 tablet_loader.cc:96] loaded metadata for tablet ddb06adb0b7d4f888e489eb5721321fc (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089])
I20250625 08:04:35.660256 2673 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:35.660888 2673 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:35.660938 2673 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:35.661613 2673 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:35.662108 2673 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Loaded TSK: 0
I20250625 08:04:35.662889 2673 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:35.663400 2673 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:35.663874 2673 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 20
I20250625 08:04:35.695513 2602 master_service.cc:948] ReplaceTablet: received request to replace tablet bcdaf64470204ff4986224f728a51ac8 from {username='slave'} at 127.0.0.1:50526
I20250625 08:04:35.695956 2601 master_service.cc:948] ReplaceTablet: received request to replace tablet 18ea19fdd2814cc08c6315b3463bd3a9 from {username='slave'} at 127.0.0.1:50526
I20250625 08:04:35.696110 2597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50526:
name: "default.Table_98189dd9ad454e5d8ee6edd848927c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:35.696815 2595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50526:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:35.696952 2595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50526:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:35.697137 2594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50526:
name: "default.Table_d78071777f3646548f7f75cea674ecdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:35.697399 2593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50526:
name: "default.Table_f22b18eed5c34ffaa7af847a3c64598d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:35.697499 2600 master_service.cc:948] ReplaceTablet: received request to replace tablet 99a2e60d7c7a4c419b609aa64d8b7334 from {username='slave'} at 127.0.0.1:50526
I20250625 08:04:35.697659 2596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50526:
name: "default.Table_28db8798a1444ed2b1d46a5a3eba537b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:35.698238 2599 master_service.cc:948] ReplaceTablet: received request to replace tablet 579f822528014659a13b9dc93111624f from {username='slave'} at 127.0.0.1:50526
I20250625 08:04:35.698930 2598 master_service.cc:948] ReplaceTablet: received request to replace tablet 7342ad43df3d493b994075493c6ab8a4 from {username='slave'} at 127.0.0.1:50526
W20250625 08:04:35.700428 2585 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 -> Peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Couldn't send request to peer 755093bec4b54a9588598f814c3708e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:35.700917 2401 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Refusing update from remote peer 0404561252194ce29bf2924fd5e67e22: Log matching property violated. Preceding OpId in replica: term: 5 index: 60. Preceding OpId from leader: term: 6 index: 62. (index mismatch)
I20250625 08:04:35.701457 2671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
F20250625 08:04:35.704281 2655 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 4 out of 11 received events: NotificationEvent(eventId=21, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_31023e55a4384ac88f8415871e52d142, message={"server":"","servicePrincipal":"","db":"default","table":"table_31023e55a4384ac88f8415871e52d142","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_31023e55a4384ac88f8415871e52d142\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_31023e55a4384ac88f8415871e52d142\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_31023e55a4384ac88f8415871e52d142\",\"kudu.table_id\":\"7825ea67af2d42b68200097030b08116\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1750838669, eventType=ALTER_TABLE, dbName=default, tableName=table_7cb71508c2934e1aa091c71056ae7a01, message={"server":"","servicePrincipal":"","db":"default","table":"table_b49cbf7ee55042b39d7b5d3946a35618","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_b49cbf7ee55042b39d7b5d3946a35618\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_b49cbf7ee55042b39d7b5d3946a35618\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b49cbf7ee55042b39d7b5d3946a35618\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7cb71508c2934e1aa091c71056ae7a01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_7cb71508c2934e1aa091c71056ae7a01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7cb71508c2934e1aa091c71056ae7a01\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_d78071777f3646548f7f75cea674ecdc, message={"server":"","servicePrincipal":"","db":"default","table":"table_d78071777f3646548f7f75cea674ecdc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d78071777f3646548f7f75cea674ecdc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_d78071777f3646548f7f75cea674ecdc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d78071777f3646548f7f75cea674ecdc\",\"kudu.table_id\":\"0e43d85f40e545d9b31c5cb3e62f89a8\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=CREATE_TABLE, dbName=default, tableName=table_88079d64160a4318bf8ea1807cb3a30d, message={"server":"","servicePrincipal":"","db":"default","table":"table_88079d64160a4318bf8ea1807cb3a30d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_88079d64160a4318bf8ea1807cb3a30d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_88079d64160a4318bf8ea1807cb3a30d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_88079d64160a4318bf8ea1807cb3a30d\",\"kudu.table_id\":\"7c7e3cbdaf064697a864873ed87a21b6\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669,"files":[]}, messageFormat=json-0.2, catName=hive);
I20250625 08:04:35.704612 2675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0404561252194ce29bf2924fd5e67e22. Latest consensus state: current_term: 6 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
*** Check failure stack trace: ***
I20250625 08:04:35.704742 2675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:35.706692 2674 mvcc.cc:204] Tried to move back new op lower bound from 7171435215660253184 to 7171435215469731840. Current Snapshot: MvccSnapshot[applied={T|T < 7171435215660253184}]
I20250625 08:04:35.707625 2675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:35.707762 2675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f7e50d73dcd google::LogMessage::Fail() at ??:0
I20250625 08:04:35.708115 2671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:35.708222 2671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: LEADER
@ 0x7f7e50d77b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f7e50d737cc google::LogMessage::Flush() at ??:0
@ 0x7f7e50d73a11 google::LogMessage::~LogMessage() at ??:0
I20250625 08:04:35.716816 2601 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:35.717024 2600 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet 99a2e60d7c7a4c419b609aa64d8b7334
I20250625 08:04:35.718052 2598 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet 7342ad43df3d493b994075493c6ab8a4
I20250625 08:04:35.718326 2599 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet 579f822528014659a13b9dc93111624f
I20250625 08:04:35.718592 2600 catalog_manager.cc:6652] ReplaceTablet: tablet 99a2e60d7c7a4c419b609aa64d8b7334 deleted and replaced by tablet 9c3205b0966048aba05895ad53861642
I20250625 08:04:35.718711 2598 catalog_manager.cc:6652] ReplaceTablet: tablet 7342ad43df3d493b994075493c6ab8a4 deleted and replaced by tablet ee4801582108472c922fea326dbb21cf
I20250625 08:04:35.718905 2600 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50526:
table { table_name: "default.Table_88079d64160a4318bf8ea1807cb3a30d" } modify_external_catalogs: true
I20250625 08:04:35.718981 2598 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50526:
table { table_name: "default.Table_23c54834c68a47bd9654b34281b53e7d" } new_table_name: "default.Table_fbd9fab6fc2a480f88e12bb1ada0b9f6" modify_external_catalogs: true
I20250625 08:04:35.719069 2600 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50526:
table { table_name: "default.Table_88079d64160a4318bf8ea1807cb3a30d" } modify_external_catalogs: true
I20250625 08:04:35.716816 2602 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet bcdaf64470204ff4986224f728a51ac8
I20250625 08:04:35.719398 2602 catalog_manager.cc:6652] ReplaceTablet: tablet bcdaf64470204ff4986224f728a51ac8 deleted and replaced by tablet 05d93f5446454d50a2064fd27758aeb5
I20250625 08:04:35.719651 2602 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50526:
name: "default.Table_035cffaa28ec4a838bb218bb37230370"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:35.720193 2601 catalog_manager.cc:6652] ReplaceTablet: tablet 18ea19fdd2814cc08c6315b3463bd3a9 deleted and replaced by tablet d6b8df079002443e9a6b72c15c7c47b7
I20250625 08:04:35.720384 2601 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50526:
name: "default.Table_174089cfe43b4dbb9ed1b9c2921f9187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:35.722510 2599 catalog_manager.cc:6652] ReplaceTablet: tablet 579f822528014659a13b9dc93111624f deleted and replaced by tablet 48f3f72f9ca843298509d81fb5c4232e
I20250625 08:04:35.722755 2599 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50526:
name: "default.Table_128075ffc1db4314b52b7c28536f6e89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f7e55d0cc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 08:04:35.726996 1958 tablet_service.cc:1515] Processing DeleteTablet for tablet 99a2e60d7c7a4c419b609aa64d8b7334 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c3205b0966048aba05895ad53861642) from {username='slave'} at 127.0.0.1:35482
I20250625 08:04:35.727209 1959 tablet_service.cc:1515] Processing DeleteTablet for tablet 579f822528014659a13b9dc93111624f with delete_type TABLET_DATA_DELETED (replaced by tablet 48f3f72f9ca843298509d81fb5c4232e) from {username='slave'} at 127.0.0.1:35482
I20250625 08:04:35.727322 1960 tablet_service.cc:1515] Processing DeleteTablet for tablet 7342ad43df3d493b994075493c6ab8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4801582108472c922fea326dbb21cf) from {username='slave'} at 127.0.0.1:35482
I20250625 08:04:35.727401 1961 tablet_service.cc:1515] Processing DeleteTablet for tablet bcdaf64470204ff4986224f728a51ac8 with delete_type TABLET_DATA_DELETED (replaced by tablet 05d93f5446454d50a2064fd27758aeb5) from {username='slave'} at 127.0.0.1:35482
I20250625 08:04:35.727505 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet 18ea19fdd2814cc08c6315b3463bd3a9 with delete_type TABLET_DATA_DELETED (replaced by tablet d6b8df079002443e9a6b72c15c7c47b7) from {username='slave'} at 127.0.0.1:35482
@ 0x7f7e55d0bba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 08:04:35.731220 1962 tablet_service.cc:1468] Processing CreateTablet for tablet 9c3205b0966048aba05895ad53861642 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.731220 1960 tablet_service.cc:1468] Processing CreateTablet for tablet 05d93f5446454d50a2064fd27758aeb5 (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.731585 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c3205b0966048aba05895ad53861642. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:35.731724 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d93f5446454d50a2064fd27758aeb5. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7e55d0b30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 08:04:35.733815 1961 tablet_service.cc:1468] Processing CreateTablet for tablet d6b8df079002443e9a6b72c15c7c47b7 (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.734071 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b8df079002443e9a6b72c15c7c47b7. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7e55d10a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250625 08:04:35.735414 2696 tablet_bootstrap.cc:492] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
@ 0x55e1a658d752 std::function<>::operator()() at ??:0
I20250625 08:04:35.736166 1959 tablet_service.cc:1468] Processing CreateTablet for tablet 48f3f72f9ca843298509d81fb5c4232e (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.736385 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48f3f72f9ca843298509d81fb5c4232e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:35.736809 2696 tablet_bootstrap.cc:654] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.738198 1958 tablet_service.cc:1468] Processing CreateTablet for tablet ee4801582108472c922fea326dbb21cf (DEFAULT_TABLE table=default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.738507 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee4801582108472c922fea326dbb21cf. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7e521c50b7 kudu::Thread::SuperviseThread() at ??:0
I20250625 08:04:35.740398 1821 tablet_service.cc:1515] Processing DeleteTablet for tablet 99a2e60d7c7a4c419b609aa64d8b7334 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c3205b0966048aba05895ad53861642) from {username='slave'} at 127.0.0.1:56114
I20250625 08:04:35.740700 1822 tablet_service.cc:1515] Processing DeleteTablet for tablet 579f822528014659a13b9dc93111624f with delete_type TABLET_DATA_DELETED (replaced by tablet 48f3f72f9ca843298509d81fb5c4232e) from {username='slave'} at 127.0.0.1:56114
I20250625 08:04:35.740850 1823 tablet_service.cc:1515] Processing DeleteTablet for tablet 7342ad43df3d493b994075493c6ab8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4801582108472c922fea326dbb21cf) from {username='slave'} at 127.0.0.1:56114
I20250625 08:04:35.740926 1824 tablet_service.cc:1515] Processing DeleteTablet for tablet bcdaf64470204ff4986224f728a51ac8 with delete_type TABLET_DATA_DELETED (replaced by tablet 05d93f5446454d50a2064fd27758aeb5) from {username='slave'} at 127.0.0.1:56114
I20250625 08:04:35.741001 1825 tablet_service.cc:1515] Processing DeleteTablet for tablet 18ea19fdd2814cc08c6315b3463bd3a9 with delete_type TABLET_DATA_DELETED (replaced by tablet d6b8df079002443e9a6b72c15c7c47b7) from {username='slave'} at 127.0.0.1:56114
I20250625 08:04:35.741205 1827 tablet_service.cc:1468] Processing CreateTablet for tablet 9c3205b0966048aba05895ad53861642 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.741418 1827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c3205b0966048aba05895ad53861642. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7e529b46db start_thread at ??:0
I20250625 08:04:35.742828 1828 tablet_service.cc:1468] Processing CreateTablet for tablet 05d93f5446454d50a2064fd27758aeb5 (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.743062 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d93f5446454d50a2064fd27758aeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:35.743646 2696 tablet_bootstrap.cc:492] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:35.743752 2696 ts_tablet_manager.cc:1397] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
@ 0x7f7e4ff2a71f clone at ??:0
I20250625 08:04:35.744468 2696 raft_consensus.cc:357] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.744601 2696 raft_consensus.cc:383] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.744639 2696 raft_consensus.cc:738] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.744668 1826 tablet_service.cc:1468] Processing CreateTablet for tablet d6b8df079002443e9a6b72c15c7c47b7 (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.744781 2696 consensus_queue.cc:260] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.744895 1826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b8df079002443e9a6b72c15c7c47b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:35.745278 1819 tablet_service.cc:1468] Processing CreateTablet for tablet ee4801582108472c922fea326dbb21cf (DEFAULT_TABLE table=default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.745530 1819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee4801582108472c922fea326dbb21cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:35.745771 1820 tablet_service.cc:1468] Processing CreateTablet for tablet 48f3f72f9ca843298509d81fb5c4232e (DEFAULT_TABLE table=default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:35.746013 1820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48f3f72f9ca843298509d81fb5c4232e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:35.746043 2696 ts_tablet_manager.cc:1428] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:35.746246 2696 tablet_bootstrap.cc:492] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:35.747336 2696 tablet_bootstrap.cc:654] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.748260 2700 tablet_bootstrap.cc:492] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:35.748436 2699 tablet_replica.cc:331] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:35.748564 2699 raft_consensus.cc:2241] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:35.748680 2699 raft_consensus.cc:2270] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.748708 2695 tablet_replica.cc:331] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:35.748839 2695 raft_consensus.cc:2241] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:35.748945 2695 raft_consensus.cc:2270] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.749049 2699 ts_tablet_manager.cc:1905] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.749229 2700 tablet_bootstrap.cc:654] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.749301 2695 ts_tablet_manager.cc:1905] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.750411 2700 tablet_bootstrap.cc:492] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:35.750536 2700 ts_tablet_manager.cc:1397] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:35.751016 2700 raft_consensus.cc:357] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.751127 2700 raft_consensus.cc:383] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.751170 2700 raft_consensus.cc:738] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.751307 2700 consensus_queue.cc:260] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.751582 2700 ts_tablet_manager.cc:1428] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:35.751842 2700 tablet_bootstrap.cc:492] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:35.752241 2695 ts_tablet_manager.cc:1918] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.752318 2695 log.cc:1199] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/99a2e60d7c7a4c419b609aa64d8b7334
I20250625 08:04:35.752698 2695 ts_tablet_manager.cc:1939] T 99a2e60d7c7a4c419b609aa64d8b7334 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:35.752938 2700 tablet_bootstrap.cc:654] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.752996 2699 ts_tablet_manager.cc:1918] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.753050 2699 log.cc:1199] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/99a2e60d7c7a4c419b609aa64d8b7334
I20250625 08:04:35.753362 2695 tablet_replica.cc:331] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:35.753477 2699 ts_tablet_manager.cc:1939] T 99a2e60d7c7a4c419b609aa64d8b7334 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:35.753494 2695 raft_consensus.cc:2241] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:35.753674 2695 raft_consensus.cc:2270] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.754048 2695 ts_tablet_manager.cc:1905] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.754105 2699 tablet_replica.cc:331] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:35.754212 2699 raft_consensus.cc:2241] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:35.754309 2699 raft_consensus.cc:2270] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.754720 2699 ts_tablet_manager.cc:1905] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.755560 2696 tablet_bootstrap.cc:492] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:35.755648 2696 ts_tablet_manager.cc:1397] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
W20250625 08:04:35.755909 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:35.756138 2696 raft_consensus.cc:357] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.756243 2696 raft_consensus.cc:383] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.756278 2696 raft_consensus.cc:738] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.756398 2696 consensus_queue.cc:260] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.756539 2695 ts_tablet_manager.cc:1918] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.756606 2695 log.cc:1199] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/579f822528014659a13b9dc93111624f
I20250625 08:04:35.756858 2695 ts_tablet_manager.cc:1939] T 579f822528014659a13b9dc93111624f P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:35.757947 2700 tablet_bootstrap.cc:492] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:35.758031 2700 ts_tablet_manager.cc:1397] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 08:04:35.758142 2696 ts_tablet_manager.cc:1428] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 08:04:35.758581 2700 raft_consensus.cc:357] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.758699 2700 raft_consensus.cc:383] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.758738 2700 raft_consensus.cc:738] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.758862 2700 consensus_queue.cc:260] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.759155 2700 ts_tablet_manager.cc:1428] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:35.759375 2700 tablet_bootstrap.cc:492] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:35.759469 2696 tablet_bootstrap.cc:492] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:35.760422 2700 tablet_bootstrap.cc:654] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.760911 2696 tablet_bootstrap.cc:654] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.761752 2695 tablet_replica.cc:331] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:35.761879 2695 raft_consensus.cc:2241] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:35.761988 2695 raft_consensus.cc:2270] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.762368 2695 ts_tablet_manager.cc:1905] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.763778 2699 ts_tablet_manager.cc:1918] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.763844 2699 log.cc:1199] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/579f822528014659a13b9dc93111624f
I20250625 08:04:35.763859 2700 tablet_bootstrap.cc:492] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:35.763935 2700 ts_tablet_manager.cc:1397] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 08:04:35.764102 2699 ts_tablet_manager.cc:1939] T 579f822528014659a13b9dc93111624f P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:35.764503 2700 raft_consensus.cc:357] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.764611 2700 raft_consensus.cc:383] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.764652 2700 raft_consensus.cc:738] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.764714 2695 ts_tablet_manager.cc:1918] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.764784 2695 log.cc:1199] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/7342ad43df3d493b994075493c6ab8a4
I20250625 08:04:35.764791 2699 tablet_replica.cc:331] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:35.764775 2700 consensus_queue.cc:260] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.764900 2699 raft_consensus.cc:2241] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:35.764983 2699 raft_consensus.cc:2270] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.765040 2695 ts_tablet_manager.cc:1939] T 7342ad43df3d493b994075493c6ab8a4 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:35.765079 2700 ts_tablet_manager.cc:1428] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:35.765265 2700 tablet_bootstrap.cc:492] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:35.765323 2699 ts_tablet_manager.cc:1905] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.765733 2695 tablet_replica.cc:331] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:35.765848 2695 raft_consensus.cc:2241] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:35.765945 2695 raft_consensus.cc:2270] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.766429 2695 ts_tablet_manager.cc:1905] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.767544 2699 ts_tablet_manager.cc:1918] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.767611 2699 log.cc:1199] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/7342ad43df3d493b994075493c6ab8a4
I20250625 08:04:35.767851 2699 ts_tablet_manager.cc:1939] T 7342ad43df3d493b994075493c6ab8a4 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:35.768500 2699 tablet_replica.cc:331] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:35.768602 2695 ts_tablet_manager.cc:1918] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.768622 2699 raft_consensus.cc:2241] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:35.768678 2695 log.cc:1199] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/bcdaf64470204ff4986224f728a51ac8
I20250625 08:04:35.768718 2699 raft_consensus.cc:2270] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.768939 2695 ts_tablet_manager.cc:1939] T bcdaf64470204ff4986224f728a51ac8 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:35.769047 2699 ts_tablet_manager.cc:1905] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.769204 2700 tablet_bootstrap.cc:654] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.769630 2695 tablet_replica.cc:331] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:35.769744 2695 raft_consensus.cc:2241] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:35.769937 2695 raft_consensus.cc:2270] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.770272 2695 ts_tablet_manager.cc:1905] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.770380 2696 tablet_bootstrap.cc:492] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:35.770529 2696 ts_tablet_manager.cc:1397] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250625 08:04:35.771023 2699 ts_tablet_manager.cc:1918] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.771083 2699 log.cc:1199] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/bcdaf64470204ff4986224f728a51ac8
I20250625 08:04:35.771104 2696 raft_consensus.cc:357] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.771227 2696 raft_consensus.cc:383] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.771270 2696 raft_consensus.cc:738] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.771345 2699 ts_tablet_manager.cc:1939] T bcdaf64470204ff4986224f728a51ac8 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:35.771454 2696 consensus_queue.cc:260] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.771749 2696 ts_tablet_manager.cc:1428] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:35.771951 2696 tablet_bootstrap.cc:492] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:35.771999 2699 tablet_replica.cc:331] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:35.772087 2699 raft_consensus.cc:2241] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:35.772174 2699 raft_consensus.cc:2270] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:35.772243 2695 ts_tablet_manager.cc:1918] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.772306 2695 log.cc:1199] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:35.772526 2699 ts_tablet_manager.cc:1905] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:35.772560 2695 ts_tablet_manager.cc:1939] T 18ea19fdd2814cc08c6315b3463bd3a9 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:35.773002 2696 tablet_bootstrap.cc:654] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.774113 2696 tablet_bootstrap.cc:492] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:35.774184 2696 ts_tablet_manager.cc:1397] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:35.774694 2696 raft_consensus.cc:357] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.774792 2696 raft_consensus.cc:383] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.774827 2696 raft_consensus.cc:738] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.774937 2696 consensus_queue.cc:260] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.775209 2696 ts_tablet_manager.cc:1428] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:35.775394 2696 tablet_bootstrap.cc:492] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:35.776396 2696 tablet_bootstrap.cc:654] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.776690 2699 ts_tablet_manager.cc:1918] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:35.776774 2699 log.cc:1199] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:35.777042 2699 ts_tablet_manager.cc:1939] T 18ea19fdd2814cc08c6315b3463bd3a9 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:35.777786 2700 tablet_bootstrap.cc:492] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:35.777869 2700 ts_tablet_manager.cc:1397] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250625 08:04:35.778304 2696 tablet_bootstrap.cc:492] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:35.778376 2696 ts_tablet_manager.cc:1397] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:35.778489 2700 raft_consensus.cc:357] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.778597 2700 raft_consensus.cc:383] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.778630 2700 raft_consensus.cc:738] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.778751 2700 consensus_queue.cc:260] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.778882 2696 raft_consensus.cc:357] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.778970 2696 raft_consensus.cc:383] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.778999 2696 raft_consensus.cc:738] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.779036 2700 ts_tablet_manager.cc:1428] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 08:04:35.779162 2696 consensus_queue.cc:260] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.779238 2700 tablet_bootstrap.cc:492] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:35.779431 2696 ts_tablet_manager.cc:1428] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:35.780261 2700 tablet_bootstrap.cc:654] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:35.782603 2700 tablet_bootstrap.cc:492] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:35.782691 2700 ts_tablet_manager.cc:1397] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250625 08:04:35.783187 2700 raft_consensus.cc:357] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.783284 2700 raft_consensus.cc:383] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:35.783316 2700 raft_consensus.cc:738] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:35.783432 2700 consensus_queue.cc:260] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.783704 2700 ts_tablet_manager.cc:1428] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:35.825636 2546 raft_consensus.cc:491] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:35.825811 2546 raft_consensus.cc:513] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.826337 2546 leader_election.cc:290] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:35.826892 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b8df079002443e9a6b72c15c7c47b7" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:35.827066 1847 raft_consensus.cc:2466] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
W20250625 08:04:35.827241 1919 leader_election.cc:343] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: d6b8df079002443e9a6b72c15c7c47b7
I20250625 08:04:35.827418 1919 leader_election.cc:304] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters: eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:35.827636 2546 raft_consensus.cc:2802] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:35.827701 2546 raft_consensus.cc:491] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:35.827739 2546 raft_consensus.cc:3058] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:35.828991 2546 raft_consensus.cc:513] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.829459 2546 leader_election.cc:290] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:35.829895 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b8df079002443e9a6b72c15c7c47b7" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:35.830040 1847 raft_consensus.cc:3058] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
W20250625 08:04:35.830233 1919 leader_election.cc:343] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: d6b8df079002443e9a6b72c15c7c47b7
I20250625 08:04:35.830983 1847 raft_consensus.cc:2466] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:35.831290 1919 leader_election.cc:304] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters: eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:35.831491 2546 raft_consensus.cc:2802] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:35.831707 2546 raft_consensus.cc:695] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:35.831861 2546 consensus_queue.cc:237] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
W20250625 08:04:35.932601 1919 connection.cc:537] server connection from 127.0.0.1:35482 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:35.932562 1781 connection.cc:537] server connection from 127.0.0.1:56114 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:35.932791 1644 connection.cc:537] client connection to 127.1.19.125:38317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:35.932909 2690 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:35.719773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.1.19.65:46085)
0625 08:04:35.728507 (+ 8734us) server_negotiation.cc:206] Beginning negotiation
0625 08:04:35.728512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0625 08:04:35.728529 (+ 17us) server_negotiation.cc:407] Connection header received
0625 08:04:35.728576 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0625 08:04:35.728579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0625 08:04:35.728620 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0625 08:04:35.728698 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0625 08:04:35.729113 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0625 08:04:35.748212 (+ 19099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0625 08:04:35.932719 (+184507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32784: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":8621,"thread_start_us":92,"threads_started":1}
W20250625 08:04:35.933142 2376 connection.cc:537] server connection from 127.0.0.1:46112 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:35.933341 1916 connection.cc:537] client connection to 127.1.19.125:38317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:35.934417 2036 connection.cc:537] client connection to 127.1.19.125:38317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:35.934871 2036 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.19.125:38317)
W20250625 08:04:35.935251 2036 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.19.125:38317)
W20250625 08:04:35.935338 2667 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:35.933905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.125:38317 (local address 127.1.19.67:42927)
0625 08:04:35.934020 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:35.934034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:35.934101 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:35.934764 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":36}
W20250625 08:04:35.935364 2036 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.19.125:38317)
W20250625 08:04:35.935532 2036 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.125:38317)
W20250625 08:04:35.936018 2662 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:35.933332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.125:38317 (local address 127.1.19.66:55671)
0625 08:04:35.933531 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:35.933550 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:35.933639 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:35.935596 (+ 1957us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120}
I20250625 08:04:35.946842 2539 raft_consensus.cc:491] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:35.946991 2539 raft_consensus.cc:513] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.947460 2539 leader_election.cc:290] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:35.947954 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c3205b0966048aba05895ad53861642" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:35.948109 1982 raft_consensus.cc:2466] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
W20250625 08:04:35.948235 1784 leader_election.cc:343] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: 9c3205b0966048aba05895ad53861642
I20250625 08:04:35.948412 1784 leader_election.cc:304] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters: eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:35.948639 2539 raft_consensus.cc:2802] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:35.948709 2539 raft_consensus.cc:491] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:35.948738 2539 raft_consensus.cc:3058] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:35.949627 2539 raft_consensus.cc:513] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.949956 2539 leader_election.cc:290] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:35.950381 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c3205b0966048aba05895ad53861642" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:35.950539 1982 raft_consensus.cc:3058] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
W20250625 08:04:35.950649 1784 leader_election.cc:343] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: 9c3205b0966048aba05895ad53861642
I20250625 08:04:35.951349 1982 raft_consensus.cc:2466] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:35.951627 1784 leader_election.cc:304] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters: eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:35.951821 2539 raft_consensus.cc:2802] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:35.952028 2539 raft_consensus.cc:695] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:35.952186 2539 consensus_queue.cc:237] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.952973 2546 raft_consensus.cc:491] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:35.953054 2546 raft_consensus.cc:513] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.953423 2546 leader_election.cc:290] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:35.953807 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05d93f5446454d50a2064fd27758aeb5" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:35.953934 1847 raft_consensus.cc:2466] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
W20250625 08:04:35.954135 1919 leader_election.cc:343] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: 05d93f5446454d50a2064fd27758aeb5
I20250625 08:04:35.954260 1919 leader_election.cc:304] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters: eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:35.954406 2546 raft_consensus.cc:2802] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:35.954576 2546 raft_consensus.cc:491] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:35.954610 2546 raft_consensus.cc:3058] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:35.955335 2546 raft_consensus.cc:513] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:35.955670 2546 leader_election.cc:290] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:35.955955 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05d93f5446454d50a2064fd27758aeb5" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:35.956048 1847 raft_consensus.cc:3058] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
W20250625 08:04:35.956225 1919 leader_election.cc:343] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: 05d93f5446454d50a2064fd27758aeb5
I20250625 08:04:35.956809 1847 raft_consensus.cc:2466] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:35.957055 1919 leader_election.cc:304] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters: eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:35.957267 2546 raft_consensus.cc:2802] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
W20250625 08:04:35.957330 1647 consensus_peers.cc:487] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7342ad43df3d493b994075493c6ab8a4. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:35.957356 2546 raft_consensus.cc:695] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:35.957511 2546 consensus_queue.cc:237] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
W20250625 08:04:35.979918 1647 consensus_peers.cc:487] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcdaf64470204ff4986224f728a51ac8. This is attempt 1: this message will repeat every 5th retry.
W20250625 08:04:36.023707 1647 consensus_peers.cc:487] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcdaf64470204ff4986224f728a51ac8. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:36.035563 2546 raft_consensus.cc:491] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:36.035688 2546 raft_consensus.cc:513] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:36.036083 2546 leader_election.cc:290] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:36.036480 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48f3f72f9ca843298509d81fb5c4232e" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:36.036638 1847 raft_consensus.cc:2466] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
W20250625 08:04:36.036839 1919 leader_election.cc:343] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: 48f3f72f9ca843298509d81fb5c4232e
I20250625 08:04:36.036983 1919 leader_election.cc:304] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters: eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:36.037138 2546 raft_consensus.cc:2802] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:36.037199 2546 raft_consensus.cc:491] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:36.037264 2546 raft_consensus.cc:3058] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:36.038107 2546 raft_consensus.cc:513] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:36.038528 2546 leader_election.cc:290] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:36.038960 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48f3f72f9ca843298509d81fb5c4232e" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:36.039068 1847 raft_consensus.cc:3058] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:36.039829 1847 raft_consensus.cc:2466] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:36.040118 1919 leader_election.cc:304] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:36.040278 2546 raft_consensus.cc:2802] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
W20250625 08:04:36.040328 1919 leader_election.cc:343] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: 48f3f72f9ca843298509d81fb5c4232e
I20250625 08:04:36.040386 2546 raft_consensus.cc:695] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:36.040585 2546 consensus_queue.cc:237] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
W20250625 08:04:36.051668 1647 consensus_peers.cc:487] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7342ad43df3d493b994075493c6ab8a4. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:36.072686 2546 raft_consensus.cc:491] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:36.072834 2546 raft_consensus.cc:513] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:36.073244 2546 leader_election.cc:290] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:36.073751 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee4801582108472c922fea326dbb21cf" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:36.073909 1847 raft_consensus.cc:2466] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
W20250625 08:04:36.074010 1919 leader_election.cc:343] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: ee4801582108472c922fea326dbb21cf
I20250625 08:04:36.074153 1919 leader_election.cc:304] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters: eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:36.074293 2546 raft_consensus.cc:2802] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:36.074354 2546 raft_consensus.cc:491] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:36.074399 2546 raft_consensus.cc:3058] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:36.075325 2546 raft_consensus.cc:513] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:36.075699 2546 leader_election.cc:290] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:36.076153 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee4801582108472c922fea326dbb21cf" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:36.076308 1847 raft_consensus.cc:3058] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
W20250625 08:04:36.076395 1919 leader_election.cc:343] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Not found: Tablet not found: ee4801582108472c922fea326dbb21cf
I20250625 08:04:36.077181 1847 raft_consensus.cc:2466] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:36.077479 1919 leader_election.cc:304] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters: eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:36.077649 2546 raft_consensus.cc:2802] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:36.077741 2546 raft_consensus.cc:695] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:36.077878 2546 consensus_queue.cc:237] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
W20250625 08:04:36.217911 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:36.239732 2727 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:36.239881 2727 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:36.240370 2727 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:36.241513 2376 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:36.241605 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:36.241667 2375 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:36.241786 2727 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250625 08:04:36.259385 1919 consensus_peers.cc:487] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 -> Peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Couldn't send request to peer eebfa8472fe04080ac012e2c91bedaae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6b8df079002443e9a6b72c15c7c47b7. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:36.283922 2546 consensus_queue.cc:1035] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 08:04:36.375155 1784 consensus_peers.cc:487] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf -> Peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Couldn't send request to peer eebfa8472fe04080ac012e2c91bedaae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c3205b0966048aba05895ad53861642. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:36.479122 2539 consensus_queue.cc:1035] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 08:04:36.519186 1919 consensus_peers.cc:487] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 -> Peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Couldn't send request to peer eebfa8472fe04080ac012e2c91bedaae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee4801582108472c922fea326dbb21cf. This is attempt 1: this message will repeat every 5th retry.
W20250625 08:04:36.549786 1919 consensus_peers.cc:487] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 -> Peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Couldn't send request to peer eebfa8472fe04080ac012e2c91bedaae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05d93f5446454d50a2064fd27758aeb5. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:36.556836 2546 consensus_queue.cc:1035] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:36.558544 2546 consensus_queue.cc:1035] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 08:04:36.563239 1919 consensus_peers.cc:487] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 -> Peer eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): Couldn't send request to peer eebfa8472fe04080ac012e2c91bedaae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48f3f72f9ca843298509d81fb5c4232e. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:36.610877 2360 master_runner.cc:186] Error getting master registration for 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111),
I20250625 08:04:36.610987 2360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:36.691247 2743 consensus_queue.cc:1035] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:36.723440 2731 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:36.723588 2731 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:36.723979 2731 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:04:36.724668 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:04:36.724803 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:36.724872 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:36.725033 2731 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:36.734082 2745 ts_tablet_manager.cc:927] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: Initiating tablet copy from peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:36.734699 2745 tablet_copy_client.cc:323] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Beginning tablet copy session from remote peer at address 127.1.19.67:37005
I20250625 08:04:36.735427 2003 tablet_copy_service.cc:140] P 5dae8595b9484b57bc7ccea543f83130: Received BeginTabletCopySession request for tablet d6b8df079002443e9a6b72c15c7c47b7 from peer eebfa8472fe04080ac012e2c91bedaae ({username='slave'} at 127.1.19.65:54581)
I20250625 08:04:36.735561 2003 tablet_copy_service.cc:161] P 5dae8595b9484b57bc7ccea543f83130: Beginning new tablet copy session on tablet d6b8df079002443e9a6b72c15c7c47b7 from peer eebfa8472fe04080ac012e2c91bedaae at {username='slave'} at 127.1.19.65:54581: session id = eebfa8472fe04080ac012e2c91bedaae-d6b8df079002443e9a6b72c15c7c47b7
I20250625 08:04:36.736807 2003 tablet_copy_source_session.cc:215] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 08:04:36.738387 2745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b8df079002443e9a6b72c15c7c47b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:36.740368 2745 tablet_copy_client.cc:806] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 0 data blocks...
I20250625 08:04:36.740550 2745 tablet_copy_client.cc:670] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 1 WAL segments...
I20250625 08:04:36.742400 2745 tablet_copy_client.cc:538] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 08:04:36.743844 2745 tablet_bootstrap.cc:492] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:36.747628 2745 tablet_bootstrap.cc:492] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:36.747918 2745 tablet_bootstrap.cc:492] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap complete.
I20250625 08:04:36.748078 2745 ts_tablet_manager.cc:1397] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250625 08:04:36.748517 2745 raft_consensus.cc:357] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:36.748622 2745 raft_consensus.cc:738] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:36.748762 2745 consensus_queue.cc:260] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:36.749011 2745 ts_tablet_manager.cc:1428] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:36.749763 2003 tablet_copy_service.cc:342] P 5dae8595b9484b57bc7ccea543f83130: Request end of tablet copy session eebfa8472fe04080ac012e2c91bedaae-d6b8df079002443e9a6b72c15c7c47b7 received from {username='slave'} at 127.1.19.65:54581
I20250625 08:04:36.749890 2003 tablet_copy_service.cc:434] P 5dae8595b9484b57bc7ccea543f83130: ending tablet copy session eebfa8472fe04080ac012e2c91bedaae-d6b8df079002443e9a6b72c15c7c47b7 on tablet d6b8df079002443e9a6b72c15c7c47b7 with peer eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:36.806723 2745 ts_tablet_manager.cc:927] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: Initiating tablet copy from peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:36.807072 2745 tablet_copy_client.cc:323] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Beginning tablet copy session from remote peer at address 127.1.19.66:46071
I20250625 08:04:36.807726 1867 tablet_copy_service.cc:140] P de62034f80d9463dbd6386fc3e90fdaf: Received BeginTabletCopySession request for tablet 9c3205b0966048aba05895ad53861642 from peer eebfa8472fe04080ac012e2c91bedaae ({username='slave'} at 127.1.19.65:47169)
I20250625 08:04:36.807866 1867 tablet_copy_service.cc:161] P de62034f80d9463dbd6386fc3e90fdaf: Beginning new tablet copy session on tablet 9c3205b0966048aba05895ad53861642 from peer eebfa8472fe04080ac012e2c91bedaae at {username='slave'} at 127.1.19.65:47169: session id = eebfa8472fe04080ac012e2c91bedaae-9c3205b0966048aba05895ad53861642
I20250625 08:04:36.809027 1867 tablet_copy_source_session.cc:215] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 08:04:36.810060 2745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c3205b0966048aba05895ad53861642. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:36.811918 2745 tablet_copy_client.cc:806] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 0 data blocks...
I20250625 08:04:36.812084 2745 tablet_copy_client.cc:670] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 1 WAL segments...
I20250625 08:04:36.813520 2745 tablet_copy_client.cc:538] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 08:04:36.814718 2745 tablet_bootstrap.cc:492] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:36.817374 2745 tablet_bootstrap.cc:492] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:36.817590 2745 tablet_bootstrap.cc:492] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap complete.
I20250625 08:04:36.817732 2745 ts_tablet_manager.cc:1397] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:36.818135 2745 raft_consensus.cc:357] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:36.818249 2745 raft_consensus.cc:738] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:36.818370 2745 consensus_queue.cc:260] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:36.818651 2745 ts_tablet_manager.cc:1428] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 08:04:36.819623 1867 tablet_copy_service.cc:342] P de62034f80d9463dbd6386fc3e90fdaf: Request end of tablet copy session eebfa8472fe04080ac012e2c91bedaae-9c3205b0966048aba05895ad53861642 received from {username='slave'} at 127.1.19.65:47169
I20250625 08:04:36.819734 1867 tablet_copy_service.cc:434] P de62034f80d9463dbd6386fc3e90fdaf: ending tablet copy session eebfa8472fe04080ac012e2c91bedaae-9c3205b0966048aba05895ad53861642 on tablet 9c3205b0966048aba05895ad53861642 with peer eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:36.917598 2750 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:36.917735 2750 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:36.918164 2750 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:36.919236 2376 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:36.919385 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:36.919466 2375 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:36.919611 2750 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250625 08:04:37.030388 2745 ts_tablet_manager.cc:927] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: Initiating tablet copy from peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:37.030843 2745 tablet_copy_client.cc:323] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Beginning tablet copy session from remote peer at address 127.1.19.67:37005
I20250625 08:04:37.031328 2003 tablet_copy_service.cc:140] P 5dae8595b9484b57bc7ccea543f83130: Received BeginTabletCopySession request for tablet ee4801582108472c922fea326dbb21cf from peer eebfa8472fe04080ac012e2c91bedaae ({username='slave'} at 127.1.19.65:54581)
I20250625 08:04:37.031426 2003 tablet_copy_service.cc:161] P 5dae8595b9484b57bc7ccea543f83130: Beginning new tablet copy session on tablet ee4801582108472c922fea326dbb21cf from peer eebfa8472fe04080ac012e2c91bedaae at {username='slave'} at 127.1.19.65:54581: session id = eebfa8472fe04080ac012e2c91bedaae-ee4801582108472c922fea326dbb21cf
I20250625 08:04:37.032212 2003 tablet_copy_source_session.cc:215] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 08:04:37.032945 2745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee4801582108472c922fea326dbb21cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:37.034695 2745 tablet_copy_client.cc:806] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 0 data blocks...
I20250625 08:04:37.034838 2745 tablet_copy_client.cc:670] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 1 WAL segments...
I20250625 08:04:37.035827 2745 tablet_copy_client.cc:538] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 08:04:37.036839 2745 tablet_bootstrap.cc:492] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:37.039630 2745 tablet_bootstrap.cc:492] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:37.039839 2745 tablet_bootstrap.cc:492] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: Bootstrap complete.
I20250625 08:04:37.039973 2745 ts_tablet_manager.cc:1397] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20250625 08:04:37.040391 2745 raft_consensus.cc:357] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:37.040484 2745 raft_consensus.cc:738] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:37.040632 2745 consensus_queue.cc:260] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:37.041426 2745 ts_tablet_manager.cc:1428] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 08:04:37.041877 2003 tablet_copy_service.cc:342] P 5dae8595b9484b57bc7ccea543f83130: Request end of tablet copy session eebfa8472fe04080ac012e2c91bedaae-ee4801582108472c922fea326dbb21cf received from {username='slave'} at 127.1.19.65:54581
I20250625 08:04:37.041977 2003 tablet_copy_service.cc:434] P 5dae8595b9484b57bc7ccea543f83130: ending tablet copy session eebfa8472fe04080ac012e2c91bedaae-ee4801582108472c922fea326dbb21cf on tablet ee4801582108472c922fea326dbb21cf with peer eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:37.068044 2745 ts_tablet_manager.cc:927] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: Initiating tablet copy from peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:37.068481 2745 tablet_copy_client.cc:323] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Beginning tablet copy session from remote peer at address 127.1.19.67:37005
I20250625 08:04:37.069238 2003 tablet_copy_service.cc:140] P 5dae8595b9484b57bc7ccea543f83130: Received BeginTabletCopySession request for tablet 05d93f5446454d50a2064fd27758aeb5 from peer eebfa8472fe04080ac012e2c91bedaae ({username='slave'} at 127.1.19.65:54581)
I20250625 08:04:37.069350 2003 tablet_copy_service.cc:161] P 5dae8595b9484b57bc7ccea543f83130: Beginning new tablet copy session on tablet 05d93f5446454d50a2064fd27758aeb5 from peer eebfa8472fe04080ac012e2c91bedaae at {username='slave'} at 127.1.19.65:54581: session id = eebfa8472fe04080ac012e2c91bedaae-05d93f5446454d50a2064fd27758aeb5
I20250625 08:04:37.070103 2003 tablet_copy_source_session.cc:215] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 08:04:37.070809 2745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d93f5446454d50a2064fd27758aeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:37.072439 2745 tablet_copy_client.cc:806] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 0 data blocks...
I20250625 08:04:37.072582 2745 tablet_copy_client.cc:670] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 1 WAL segments...
I20250625 08:04:37.073836 2745 tablet_copy_client.cc:538] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 08:04:37.075052 2745 tablet_bootstrap.cc:492] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:37.077796 2745 tablet_bootstrap.cc:492] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:37.077982 2745 tablet_bootstrap.cc:492] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap complete.
I20250625 08:04:37.078099 2745 ts_tablet_manager.cc:1397] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:37.078568 2745 raft_consensus.cc:357] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:37.078668 2745 raft_consensus.cc:738] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:37.078778 2745 consensus_queue.cc:260] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:37.079396 2745 ts_tablet_manager.cc:1428] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 08:04:37.079945 2003 tablet_copy_service.cc:342] P 5dae8595b9484b57bc7ccea543f83130: Request end of tablet copy session eebfa8472fe04080ac012e2c91bedaae-05d93f5446454d50a2064fd27758aeb5 received from {username='slave'} at 127.1.19.65:54581
I20250625 08:04:37.080034 2003 tablet_copy_service.cc:434] P 5dae8595b9484b57bc7ccea543f83130: ending tablet copy session eebfa8472fe04080ac012e2c91bedaae-05d93f5446454d50a2064fd27758aeb5 on tablet 05d93f5446454d50a2064fd27758aeb5 with peer eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:37.105918 2745 ts_tablet_manager.cc:927] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: Initiating tablet copy from peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:37.106235 2745 tablet_copy_client.cc:323] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Beginning tablet copy session from remote peer at address 127.1.19.67:37005
I20250625 08:04:37.106736 2003 tablet_copy_service.cc:140] P 5dae8595b9484b57bc7ccea543f83130: Received BeginTabletCopySession request for tablet 48f3f72f9ca843298509d81fb5c4232e from peer eebfa8472fe04080ac012e2c91bedaae ({username='slave'} at 127.1.19.65:54581)
I20250625 08:04:37.106855 2003 tablet_copy_service.cc:161] P 5dae8595b9484b57bc7ccea543f83130: Beginning new tablet copy session on tablet 48f3f72f9ca843298509d81fb5c4232e from peer eebfa8472fe04080ac012e2c91bedaae at {username='slave'} at 127.1.19.65:54581: session id = eebfa8472fe04080ac012e2c91bedaae-48f3f72f9ca843298509d81fb5c4232e
I20250625 08:04:37.107616 2003 tablet_copy_source_session.cc:215] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 08:04:37.108358 2745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48f3f72f9ca843298509d81fb5c4232e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:37.110107 2745 tablet_copy_client.cc:806] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 0 data blocks...
I20250625 08:04:37.110250 2745 tablet_copy_client.cc:670] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Starting download of 1 WAL segments...
I20250625 08:04:37.111285 2745 tablet_copy_client.cc:538] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 08:04:37.112278 2745 tablet_bootstrap.cc:492] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:37.115020 2745 tablet_bootstrap.cc:492] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:37.115252 2745 tablet_bootstrap.cc:492] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: Bootstrap complete.
I20250625 08:04:37.115368 2745 ts_tablet_manager.cc:1397] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250625 08:04:37.115785 2745 raft_consensus.cc:357] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:37.115876 2745 raft_consensus.cc:738] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:37.115973 2745 consensus_queue.cc:260] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:37.116389 2745 ts_tablet_manager.cc:1428] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 08:04:37.117069 2003 tablet_copy_service.cc:342] P 5dae8595b9484b57bc7ccea543f83130: Request end of tablet copy session eebfa8472fe04080ac012e2c91bedaae-48f3f72f9ca843298509d81fb5c4232e received from {username='slave'} at 127.1.19.65:54581
I20250625 08:04:37.117169 2003 tablet_copy_service.cc:434] P 5dae8595b9484b57bc7ccea543f83130: ending tablet copy session eebfa8472fe04080ac012e2c91bedaae-48f3f72f9ca843298509d81fb5c4232e on tablet 48f3f72f9ca843298509d81fb5c4232e with peer eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:37.172895 2731 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:37.173063 2731 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:37.173497 2731 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:04:37.174168 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:04:37.174293 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:04:37.174335 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:37.174491 2731 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:37.244683 1711 raft_consensus.cc:1215] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250625 08:04:37.271238 1711 raft_consensus.cc:1215] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250625 08:04:37.504181 1711 raft_consensus.cc:1215] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250625 08:04:37.639693 1711 raft_consensus.cc:1215] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250625 08:04:37.653949 1711 raft_consensus.cc:1215] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250625 08:04:37.925613 2770 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:37.925807 2770 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:37.926296 2770 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:37.926908 2375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250625 08:04:37.927850 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:37.928380 2376 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:37.928463 2376 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:37.928730 2770 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250625 08:04:38.071355 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.125:38317
--webserver_interface=127.1.19.125
--webserver_port=39497
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 08:04:38.187371 2773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:38.187623 2773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:38.187664 2773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:38.191582 2773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:38.191653 2773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:38.191676 2773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:38.191704 2773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:38.195971 2773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.125:38317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.19.125
--webserver_port=39497
--never_fsync=true
--heap_profile_path=/tmp/kudu.2773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:38.197190 2773 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:38.198359 2773 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:38.205755 2781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:38.205782 2783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:38.205761 2780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:38.206053 2773 server_base.cc:1048] running on GCE node
I20250625 08:04:38.206785 2773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:38.207830 2773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:38.209004 2773 hybrid_clock.cc:648] HybridClock initialized: now 1750838678208991 us; error 32 us; skew 500 ppm
I20250625 08:04:38.215401 2773 webserver.cc:469] Webserver started at http://127.1.19.125:39497/ using document root <none> and password file <none>
I20250625 08:04:38.216183 2773 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:38.216275 2773 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:38.221333 2773 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250625 08:04:38.223814 2789 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:38.225009 2773 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250625 08:04:38.225128 2773 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
uuid: "0404561252194ce29bf2924fd5e67e22"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:38.225556 2773 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:38.255852 2773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:38.256542 2773 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:38.256702 2773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:38.264062 2773 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.125:38317
I20250625 08:04:38.264068 2843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.125:38317 every 8 connection(s)
I20250625 08:04:38.265303 2773 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
I20250625 08:04:38.268587 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 2773
I20250625 08:04:38.269515 2844 sys_catalog.cc:263] Verifying existing consensus state
I20250625 08:04:38.271497 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap starting.
I20250625 08:04:38.281559 2844 log.cc:826] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:38.316323 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap replayed 1/1 log segments. Stats: ops{read=68 overwritten=1 applied=67 ignored=0} inserts{seen=43 ignored=0} mutations{seen=52 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:38.316782 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap complete.
I20250625 08:04:38.319550 2844 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:38.319911 2844 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0404561252194ce29bf2924fd5e67e22, State: Initialized, Role: FOLLOWER
I20250625 08:04:38.320356 2844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 67, Last appended: 6.67, Last appended by leader: 67, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:38.320858 2848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:38.320984 2848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:38.321542 2844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:38.328580 2862 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:38.328672 2862 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: loading cluster ID for follower catalog manager: success
I20250625 08:04:38.332559 2862 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: acquiring CA information for follower catalog manager: success
I20250625 08:04:38.334774 2862 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:38.350862 2773 master_runner.cc:186] Error getting master registration for 127.1.19.124:37745: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111),
I20250625 08:04:38.352471 2773 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:38.377866 1759 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:38.380108 2807 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" instance_seqno: 1750838666815429) as {username='slave'} at 127.1.19.65:43383; Asking this server to re-register.
I20250625 08:04:38.380689 1759 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:38.382169 2807 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:38.469884 2731 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:38.470046 2731 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:38.470556 2731 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:04:38.471319 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:04:38.471462 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:38.471515 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:38.471745 2731 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:38.659511 2031 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:38.660727 2807 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" instance_seqno: 1750838667205184) as {username='slave'} at 127.1.19.67:60983; Asking this server to re-register.
I20250625 08:04:38.661182 2031 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:38.662086 2807 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:38.745797 2661 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:38.745958 2661 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:38.746419 2661 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:04:38.747259 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:04:38.747606 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:04:38.747666 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:38.747908 2661 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:38.751674 1896 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:38.752846 2807 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" instance_seqno: 1750838667017582) as {username='slave'} at 127.1.19.66:34693; Asking this server to re-register.
I20250625 08:04:38.753319 1896 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:38.754205 2807 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:38.934376 2873 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:38.934638 2873 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:38.935873 2873 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:38.936379 2401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 7 candidate_status { last_received { term: 6 index: 67 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
I20250625 08:04:38.936532 2401 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0404561252194ce29bf2924fd5e67e22 in term 6.
W20250625 08:04:38.936579 2791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:38.936923 2793 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters:
I20250625 08:04:38.937171 2873 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250625 08:04:38.937266 2873 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:38.937325 2873 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 6 FOLLOWER]: Advancing to term 7
W20250625 08:04:38.937557 2791 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:38.939432 2873 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:38.939828 2873 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 7 election: Requested vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:38.940322 2401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 7 candidate_status { last_received { term: 6 index: 67 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
I20250625 08:04:38.940464 2401 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 6 FOLLOWER]: Advancing to term 7
W20250625 08:04:38.940896 2791 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:38.941479 2401 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0404561252194ce29bf2924fd5e67e22 in term 7.
I20250625 08:04:38.941835 2793 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 755093bec4b54a9588598f814c3708e1
I20250625 08:04:38.942034 2873 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 7 FOLLOWER]: Leader election won for term 7
I20250625 08:04:38.942231 2873 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 7 LEADER]: Becoming Leader. State: Replica: 0404561252194ce29bf2924fd5e67e22, State: Running, Role: LEADER
I20250625 08:04:38.942612 2873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 6.67, Last appended by leader: 67, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:38.944494 2876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0404561252194ce29bf2924fd5e67e22. Latest consensus state: current_term: 7 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:38.944656 2876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:38.945010 2878 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:38.946827 2878 catalog_manager.cc:671] Loaded metadata for table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]
I20250625 08:04:38.947149 2878 catalog_manager.cc:671] Loaded metadata for table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]
I20250625 08:04:38.947379 2878 catalog_manager.cc:671] Loaded metadata for table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]
I20250625 08:04:38.947595 2878 catalog_manager.cc:671] Loaded metadata for table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]
I20250625 08:04:38.947813 2878 catalog_manager.cc:671] Loaded metadata for table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]
I20250625 08:04:38.948019 2878 catalog_manager.cc:671] Loaded metadata for table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]
I20250625 08:04:38.948230 2878 catalog_manager.cc:671] Loaded metadata for table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]
I20250625 08:04:38.948455 2878 catalog_manager.cc:671] Loaded metadata for table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]
I20250625 08:04:38.949707 2878 tablet_loader.cc:96] loaded metadata for tablet 05d93f5446454d50a2064fd27758aeb5 (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a])
I20250625 08:04:38.950052 2878 tablet_loader.cc:96] loaded metadata for tablet 1564b8dab2e9429f9c2a98f0935928f5 (table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d])
I20250625 08:04:38.950224 2878 tablet_loader.cc:96] loaded metadata for tablet 17e0682e2bfe43b7a445f595d8929034 (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6])
I20250625 08:04:38.950798 2878 tablet_loader.cc:96] loaded metadata for tablet 48f3f72f9ca843298509d81fb5c4232e (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020])
I20250625 08:04:38.951283 2878 tablet_loader.cc:96] loaded metadata for tablet 9c3205b0966048aba05895ad53861642 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875])
I20250625 08:04:38.952107 2878 tablet_loader.cc:96] loaded metadata for tablet d6b8df079002443e9a6b72c15c7c47b7 (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206])
I20250625 08:04:38.952268 2878 tablet_loader.cc:96] loaded metadata for tablet ddb06adb0b7d4f888e489eb5721321fc (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089])
I20250625 08:04:38.952438 2878 tablet_loader.cc:96] loaded metadata for tablet ee4801582108472c922fea326dbb21cf (table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904])
I20250625 08:04:38.952677 2878 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:38.953337 2878 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:38.953389 2878 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:38.954113 2878 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:38.954725 2878 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Loaded TSK: 0
I20250625 08:04:38.955646 2878 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:38.956444 2878 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:38.957121 2878 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 20
I20250625 08:04:38.973825 2807 master_service.cc:948] ReplaceTablet: received request to replace tablet 17e0682e2bfe43b7a445f595d8929034 from {username='slave'} at 127.0.0.1:51254
I20250625 08:04:38.974326 2803 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51254:
table { table_name: "default.Table_a3a8c500599e482da5b7120ec26a797a" } new_table_name: "default.Table_6894ca710ded4040ba337309d454d090" modify_external_catalogs: true
I20250625 08:04:38.974750 2802 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51254:
table { table_name: "default.Table_5eebaa7c05694491963fce48525e4681" } new_table_name: "default.Table_69a1c4361ab540a48fd24a5198251c2c" modify_external_catalogs: true
I20250625 08:04:38.974699 2800 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51254:
name: "default.Table_31023e55a4384ac88f8415871e52d142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:38.974987 2799 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51254:
table { table_name: "default.Table_b49cbf7ee55042b39d7b5d3946a35618" } new_table_name: "default.Table_7cb71508c2934e1aa091c71056ae7a01" modify_external_catalogs: true
I20250625 08:04:38.975080 2808 master_service.cc:948] ReplaceTablet: received request to replace tablet 9c3205b0966048aba05895ad53861642 from {username='slave'} at 127.0.0.1:51254
I20250625 08:04:38.975538 2801 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51254:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:38.975649 2801 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51254:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:38.975785 2804 master_service.cc:948] ReplaceTablet: received request to replace tablet d6b8df079002443e9a6b72c15c7c47b7 from {username='slave'} at 127.0.0.1:51254
I20250625 08:04:38.976233 2806 master_service.cc:948] ReplaceTablet: received request to replace tablet 05d93f5446454d50a2064fd27758aeb5 from {username='slave'} at 127.0.0.1:51254
I20250625 08:04:38.976444 2805 master_service.cc:948] ReplaceTablet: received request to replace tablet 17e0682e2bfe43b7a445f595d8929034 from {username='slave'} at 127.0.0.1:51254
I20250625 08:04:38.979643 2401 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Refusing update from remote peer 0404561252194ce29bf2924fd5e67e22: Log matching property violated. Preceding OpId in replica: term: 6 index: 67. Preceding OpId from leader: term: 7 index: 69. (index mismatch)
I20250625 08:04:38.980198 2876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250625 08:04:38.981809 1686 tablet_service.cc:1515] Processing DeleteTablet for tablet bcdaf64470204ff4986224f728a51ac8 with delete_type TABLET_DATA_DELETED (replaced by tablet 05d93f5446454d50a2064fd27758aeb5) from {username='slave'} at 127.0.0.1:32794
W20250625 08:04:38.982497 2791 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 -> Peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Couldn't send request to peer 755093bec4b54a9588598f814c3708e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:38.982666 2883 tablet_replica.cc:331] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
F20250625 08:04:38.982803 2861 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 4 out of 11 received events: NotificationEvent(eventId=21, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_31023e55a4384ac88f8415871e52d142, message={"server":"","servicePrincipal":"","db":"default","table":"table_31023e55a4384ac88f8415871e52d142","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_31023e55a4384ac88f8415871e52d142\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_31023e55a4384ac88f8415871e52d142\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_31023e55a4384ac88f8415871e52d142\",\"kudu.table_id\":\"7825ea67af2d42b68200097030b08116\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1750838669, eventType=ALTER_TABLE, dbName=default, tableName=table_7cb71508c2934e1aa091c71056ae7a01, message={"server":"","servicePrincipal":"","db":"default","table":"table_b49cbf7ee55042b39d7b5d3946a35618","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_b49cbf7ee55042b39d7b5d3946a35618\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_b49cbf7ee55042b39d7b5d3946a35618\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b49cbf7ee55042b39d7b5d3946a35618\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7cb71508c2934e1aa091c71056ae7a01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_7cb71508c2934e1aa091c71056ae7a01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7cb71508c2934e1aa091c71056ae7a01\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_d78071777f3646548f7f75cea674ecdc, message={"server":"","servicePrincipal":"","db":"default","table":"table_d78071777f3646548f7f75cea674ecdc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d78071777f3646548f7f75cea674ecdc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_d78071777f3646548f7f75cea674ecdc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d78071777f3646548f7f75cea674ecdc\",\"kudu.table_id\":\"0e43d85f40e545d9b31c5cb3e62f89a8\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=CREATE_TABLE, dbName=default, tableName=table_88079d64160a4318bf8ea1807cb3a30d, message={"server":"","servicePrincipal":"","db":"default","table":"table_88079d64160a4318bf8ea1807cb3a30d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_88079d64160a4318bf8ea1807cb3a30d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_88079d64160a4318bf8ea1807cb3a30d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_88079d64160a4318bf8ea1807cb3a30d\",\"kudu.table_id\":\"7c7e3cbdaf064697a864873ed87a21b6\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669,"files":[]}, messageFormat=json-0.2, catName=hive);
I20250625 08:04:38.982828 2883 raft_consensus.cc:2241] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:38.982841 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0404561252194ce29bf2924fd5e67e22. Latest consensus state: current_term: 7 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:38.982952 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:38.983079 2883 raft_consensus.cc:2270] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
*** Check failure stack trace: ***
I20250625 08:04:38.983471 2883 ts_tablet_manager.cc:1905] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:38.985812 2883 ts_tablet_manager.cc:1918] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:38.985993 2883 log.cc:1199] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/bcdaf64470204ff4986224f728a51ac8
@ 0x7efe6b819dcd google::LogMessage::Fail() at ??:0
I20250625 08:04:38.986581 2883 ts_tablet_manager.cc:1939] T bcdaf64470204ff4986224f728a51ac8 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:38.987112 2876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:38.987254 2876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:38.987414 2793 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet bcdaf64470204ff4986224f728a51ac8 (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]) successfully deleted
I20250625 08:04:38.988390 2880 mvcc.cc:204] Tried to move back new op lower bound from 7171435229097017344 to 7171435228951490560. Current Snapshot: MvccSnapshot[applied={T|T < 7171435229086846976}]
I20250625 08:04:38.988559 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "0404561252194ce29bf2924fd5e67e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:38.988667 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7efe6b81db93 google::LogMessage::SendToLog() at ??:0
I20250625 08:04:38.989187 2808 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet 9c3205b0966048aba05895ad53861642
I20250625 08:04:38.989303 2806 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet 05d93f5446454d50a2064fd27758aeb5
I20250625 08:04:38.989753 2804 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet d6b8df079002443e9a6b72c15c7c47b7
I20250625 08:04:38.990031 2806 catalog_manager.cc:6652] ReplaceTablet: tablet 05d93f5446454d50a2064fd27758aeb5 deleted and replaced by tablet 42cc323e9e8a4b6f9f58b577c9057d2a
I20250625 08:04:38.990027 1686 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c3205b0966048aba05895ad53861642 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fb1337f4e44450bbf1379ecddae8bcf) from {username='slave'} at 127.0.0.1:32794
I20250625 08:04:38.990052 2804 catalog_manager.cc:6652] ReplaceTablet: tablet d6b8df079002443e9a6b72c15c7c47b7 deleted and replaced by tablet ee517261551246b8bd14f8bb2460895c
I20250625 08:04:38.990278 2808 catalog_manager.cc:6652] ReplaceTablet: tablet 9c3205b0966048aba05895ad53861642 deleted and replaced by tablet 2fb1337f4e44450bbf1379ecddae8bcf
I20250625 08:04:38.990504 2883 tablet_replica.cc:331] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:38.990411 2804 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51254:
name: "default.Table_bd43623a5f9f40c58ad750234777ba9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:38.990635 2883 raft_consensus.cc:2241] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:38.990737 2883 raft_consensus.cc:2270] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:38.990769 2805 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51254:
name: "default.Table_28db8798a1444ed2b1d46a5a3eba537b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:38.991057 2883 ts_tablet_manager.cc:1905] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7efe6b8197cc google::LogMessage::Flush() at ??:0
I20250625 08:04:38.992867 2883 ts_tablet_manager.cc:1918] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:38.992921 2883 log.cc:1199] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/9c3205b0966048aba05895ad53861642
I20250625 08:04:38.993105 2883 ts_tablet_manager.cc:1939] T 9c3205b0966048aba05895ad53861642 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
@ 0x7efe6b819a11 google::LogMessage::~LogMessage() at ??:0
I20250625 08:04:38.995884 2793 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 9c3205b0966048aba05895ad53861642 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:38.996723 1686 tablet_service.cc:1515] Processing DeleteTablet for tablet 05d93f5446454d50a2064fd27758aeb5 with delete_type TABLET_DATA_DELETED (replaced by tablet 42cc323e9e8a4b6f9f58b577c9057d2a) from {username='slave'} at 127.0.0.1:32794
I20250625 08:04:38.989198 2807 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Sending DeleteTablet for 3 replicas of tablet 17e0682e2bfe43b7a445f595d8929034
I20250625 08:04:38.996759 1687 tablet_service.cc:1515] Processing DeleteTablet for tablet d6b8df079002443e9a6b72c15c7c47b7 with delete_type TABLET_DATA_DELETED (replaced by tablet ee517261551246b8bd14f8bb2460895c) from {username='slave'} at 127.0.0.1:32794
I20250625 08:04:38.996925 2883 tablet_replica.cc:331] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:38.997089 2883 raft_consensus.cc:2241] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:38.997161 2807 catalog_manager.cc:6652] ReplaceTablet: tablet 17e0682e2bfe43b7a445f595d8929034 deleted and replaced by tablet b97c4e286a5040b1ad425242d9cb2dea
I20250625 08:04:38.997224 2883 raft_consensus.cc:2270] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:38.997422 2807 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51254:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:38.997602 2807 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51254:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:38.997687 2883 ts_tablet_manager.cc:1905] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:38.998314 2806 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51254:
name: "default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:38.998613 2808 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51254:
name: "default.Table_98189dd9ad454e5d8ee6edd848927c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:38.999280 1686 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e0682e2bfe43b7a445f595d8929034 with delete_type TABLET_DATA_DELETED (replaced by tablet b97c4e286a5040b1ad425242d9cb2dea) from {username='slave'} at 127.0.0.1:32794
@ 0x7efe707b2c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 08:04:39.000192 2883 ts_tablet_manager.cc:1918] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.000259 2883 log.cc:1199] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/d6b8df079002443e9a6b72c15c7c47b7
I20250625 08:04:39.000537 2883 ts_tablet_manager.cc:1939] T d6b8df079002443e9a6b72c15c7c47b7 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:39.003057 2883 tablet_replica.cc:331] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:39.003203 2883 raft_consensus.cc:2241] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:39.003314 2883 raft_consensus.cc:2270] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.003549 2793 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet d6b8df079002443e9a6b72c15c7c47b7 (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]) successfully deleted
I20250625 08:04:39.003767 2883 ts_tablet_manager.cc:1905] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7efe707b1ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 08:04:39.005885 1686 tablet_service.cc:1468] Processing CreateTablet for tablet 2fb1337f4e44450bbf1379ecddae8bcf (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.006083 2883 ts_tablet_manager.cc:1918] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.006145 2883 log.cc:1199] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/05d93f5446454d50a2064fd27758aeb5
I20250625 08:04:39.006172 1686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb1337f4e44450bbf1379ecddae8bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.006403 2883 ts_tablet_manager.cc:1939] T 05d93f5446454d50a2064fd27758aeb5 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:39.007079 2883 tablet_replica.cc:331] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:39.007218 2883 raft_consensus.cc:2241] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:39.007320 2883 raft_consensus.cc:2270] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.007697 2883 ts_tablet_manager.cc:1905] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7efe707b130b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 08:04:39.008826 2793 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 05d93f5446454d50a2064fd27758aeb5 (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]) successfully deleted
I20250625 08:04:39.009615 1687 tablet_service.cc:1468] Processing CreateTablet for tablet b97c4e286a5040b1ad425242d9cb2dea (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.009739 2883 ts_tablet_manager.cc:1918] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.009791 2883 log.cc:1199] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/17e0682e2bfe43b7a445f595d8929034
I20250625 08:04:39.009886 1687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b97c4e286a5040b1ad425242d9cb2dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.010039 2883 ts_tablet_manager.cc:1939] T 17e0682e2bfe43b7a445f595d8929034 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:39.010071 1820 tablet_service.cc:1515] Processing DeleteTablet for tablet 05d93f5446454d50a2064fd27758aeb5 with delete_type TABLET_DATA_DELETED (replaced by tablet 42cc323e9e8a4b6f9f58b577c9057d2a) from {username='slave'} at 127.0.0.1:56124
I20250625 08:04:39.010421 1819 tablet_service.cc:1515] Processing DeleteTablet for tablet d6b8df079002443e9a6b72c15c7c47b7 with delete_type TABLET_DATA_DELETED (replaced by tablet ee517261551246b8bd14f8bb2460895c) from {username='slave'} at 127.0.0.1:56124
I20250625 08:04:39.010598 1826 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c3205b0966048aba05895ad53861642 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fb1337f4e44450bbf1379ecddae8bcf) from {username='slave'} at 127.0.0.1:56124
I20250625 08:04:39.010712 1828 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e0682e2bfe43b7a445f595d8929034 with delete_type TABLET_DATA_DELETED (replaced by tablet b97c4e286a5040b1ad425242d9cb2dea) from {username='slave'} at 127.0.0.1:56124
I20250625 08:04:39.010999 1827 tablet_service.cc:1468] Processing CreateTablet for tablet 2fb1337f4e44450bbf1379ecddae8bcf (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.011273 1827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb1337f4e44450bbf1379ecddae8bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.009629 1685 tablet_service.cc:1468] Processing CreateTablet for tablet 42cc323e9e8a4b6f9f58b577c9057d2a (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.011624 1685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cc323e9e8a4b6f9f58b577c9057d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.013060 2793 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 17e0682e2bfe43b7a445f595d8929034 (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]) successfully deleted
I20250625 08:04:39.013221 1686 tablet_service.cc:1468] Processing CreateTablet for tablet ee517261551246b8bd14f8bb2460895c (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.013420 1686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee517261551246b8bd14f8bb2460895c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.014019 1825 tablet_service.cc:1468] Processing CreateTablet for tablet ee517261551246b8bd14f8bb2460895c (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.014281 2901 tablet_bootstrap.cc:492] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:39.014513 1824 tablet_service.cc:1468] Processing CreateTablet for tablet b97c4e286a5040b1ad425242d9cb2dea (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.014873 1823 tablet_service.cc:1468] Processing CreateTablet for tablet 42cc323e9e8a4b6f9f58b577c9057d2a (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.014264 1825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee517261551246b8bd14f8bb2460895c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.015233 1823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cc323e9e8a4b6f9f58b577c9057d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.017408 2901 tablet_bootstrap.cc:654] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:39.018941 1824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b97c4e286a5040b1ad425242d9cb2dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.019027 1958 tablet_service.cc:1515] Processing DeleteTablet for tablet 05d93f5446454d50a2064fd27758aeb5 with delete_type TABLET_DATA_DELETED (replaced by tablet 42cc323e9e8a4b6f9f58b577c9057d2a) from {username='slave'} at 127.0.0.1:35484
I20250625 08:04:39.019294 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e0682e2bfe43b7a445f595d8929034 with delete_type TABLET_DATA_DELETED (replaced by tablet b97c4e286a5040b1ad425242d9cb2dea) from {username='slave'} at 127.0.0.1:35484
I20250625 08:04:39.019552 1962 tablet_service.cc:1468] Processing CreateTablet for tablet b97c4e286a5040b1ad425242d9cb2dea (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.019616 1959 tablet_service.cc:1515] Processing DeleteTablet for tablet d6b8df079002443e9a6b72c15c7c47b7 with delete_type TABLET_DATA_DELETED (replaced by tablet ee517261551246b8bd14f8bb2460895c) from {username='slave'} at 127.0.0.1:35484
I20250625 08:04:39.019686 1958 tablet_service.cc:1468] Processing CreateTablet for tablet ee517261551246b8bd14f8bb2460895c (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.019768 1961 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c3205b0966048aba05895ad53861642 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fb1337f4e44450bbf1379ecddae8bcf) from {username='slave'} at 127.0.0.1:35484
I20250625 08:04:39.019866 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b97c4e286a5040b1ad425242d9cb2dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.019552 1960 tablet_service.cc:1468] Processing CreateTablet for tablet 2fb1337f4e44450bbf1379ecddae8bcf (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.020025 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb1337f4e44450bbf1379ecddae8bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.019991 1963 tablet_service.cc:1468] Processing CreateTablet for tablet 42cc323e9e8a4b6f9f58b577c9057d2a (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:39.020229 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cc323e9e8a4b6f9f58b577c9057d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.021246 2904 tablet_replica.cc:331] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:39.021461 2904 raft_consensus.cc:2241] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:39.021704 2904 raft_consensus.cc:2270] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.022115 2904 ts_tablet_manager.cc:1905] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:39.022181 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee517261551246b8bd14f8bb2460895c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:39.022481 2900 tablet_replica.cc:331] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:39.022667 2900 raft_consensus.cc:2241] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:39.022781 2900 raft_consensus.cc:2270] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.022899 2901 tablet_bootstrap.cc:492] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:39.022982 2901 ts_tablet_manager.cc:1397] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 08:04:39.023099 2900 ts_tablet_manager.cc:1905] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:39.023528 2901 raft_consensus.cc:357] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.023648 2901 raft_consensus.cc:383] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.023691 2901 raft_consensus.cc:738] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.023815 2901 consensus_queue.cc:260] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.024145 2901 ts_tablet_manager.cc:1428] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:39.024317 2901 tablet_bootstrap.cc:492] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:39.026310 2901 tablet_bootstrap.cc:654] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
@ 0x7efe707b6a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250625 08:04:39.027360 2900 ts_tablet_manager.cc:1918] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.027438 2900 log.cc:1199] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/05d93f5446454d50a2064fd27758aeb5
@ 0x5555d0791752 std::function<>::operator()() at ??:0
I20250625 08:04:39.027730 2900 ts_tablet_manager.cc:1939] T 05d93f5446454d50a2064fd27758aeb5 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:39.028312 2900 tablet_replica.cc:331] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:39.028441 2900 raft_consensus.cc:2241] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:39.028549 2900 raft_consensus.cc:2270] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.028662 2793 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 05d93f5446454d50a2064fd27758aeb5 (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]) successfully deleted
I20250625 08:04:39.028913 2900 ts_tablet_manager.cc:1905] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7efe6cc6b0b7 kudu::Thread::SuperviseThread() at ??:0
I20250625 08:04:39.030758 2905 tablet_bootstrap.cc:492] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:39.031364 2899 tablet_bootstrap.cc:492] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:39.032240 2905 tablet_bootstrap.cc:654] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:39.032357 2899 tablet_bootstrap.cc:654] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
@ 0x7efe6d45a6db start_thread at ??:0
I20250625 08:04:39.033514 2900 ts_tablet_manager.cc:1918] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.033581 2900 log.cc:1199] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/d6b8df079002443e9a6b72c15c7c47b7
I20250625 08:04:39.033838 2900 ts_tablet_manager.cc:1939] T d6b8df079002443e9a6b72c15c7c47b7 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
@ 0x7efe6a9d071f clone at ??:0
I20250625 08:04:39.034471 2900 tablet_replica.cc:331] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:39.035285 2899 tablet_bootstrap.cc:492] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:39.035390 2899 ts_tablet_manager.cc:1397] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 08:04:39.035943 2899 raft_consensus.cc:357] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.036051 2899 raft_consensus.cc:383] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.036080 2899 raft_consensus.cc:738] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.036233 2899 consensus_queue.cc:260] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.037002 1759 heartbeater.cc:499] Master 127.1.19.125:38317 was elected leader, sending a full tablet report...
I20250625 08:04:39.038061 2904 ts_tablet_manager.cc:1918] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.038151 2904 log.cc:1199] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/05d93f5446454d50a2064fd27758aeb5
I20250625 08:04:39.038530 2904 ts_tablet_manager.cc:1939] T 05d93f5446454d50a2064fd27758aeb5 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:39.038774 2899 ts_tablet_manager.cc:1428] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 08:04:39.039080 2905 tablet_bootstrap.cc:492] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:39.039160 2905 ts_tablet_manager.cc:1397] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250625 08:04:39.039286 2904 tablet_replica.cc:331] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:39.039414 2904 raft_consensus.cc:2241] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:39.039616 2904 raft_consensus.cc:2270] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.040023 2904 ts_tablet_manager.cc:1905] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:39.039634 2905 raft_consensus.cc:357] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.040398 2905 raft_consensus.cc:383] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.040427 2905 raft_consensus.cc:738] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.040558 2905 consensus_queue.cc:260] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.040799 2905 ts_tablet_manager.cc:1428] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 08:04:39.040971 2905 tablet_bootstrap.cc:492] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:39.041850 2899 tablet_bootstrap.cc:492] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:39.042165 2905 tablet_bootstrap.cc:654] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:39.043092 2899 tablet_bootstrap.cc:654] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:39.043795 2900 raft_consensus.cc:2241] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:39.044054 2900 raft_consensus.cc:2270] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.044481 2900 ts_tablet_manager.cc:1905] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:39.046033 2905 tablet_bootstrap.cc:492] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:39.046123 2905 ts_tablet_manager.cc:1397] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 08:04:39.046669 2905 raft_consensus.cc:357] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.046783 2905 raft_consensus.cc:383] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.046815 2905 raft_consensus.cc:738] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.046887 2900 ts_tablet_manager.cc:1918] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.046955 2900 log.cc:1199] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/9c3205b0966048aba05895ad53861642
I20250625 08:04:39.046943 2905 consensus_queue.cc:260] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.047111 2904 ts_tablet_manager.cc:1918] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.047168 2904 log.cc:1199] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/17e0682e2bfe43b7a445f595d8929034
I20250625 08:04:39.047216 2900 ts_tablet_manager.cc:1939] T 9c3205b0966048aba05895ad53861642 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:39.047261 2905 ts_tablet_manager.cc:1428] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:39.047475 2905 tablet_bootstrap.cc:492] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:39.047513 2904 ts_tablet_manager.cc:1939] T 17e0682e2bfe43b7a445f595d8929034 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:39.048029 2904 tablet_replica.cc:331] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:39.048144 2904 raft_consensus.cc:2241] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:39.048283 2899 tablet_bootstrap.cc:492] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:39.048331 2904 raft_consensus.cc:2270] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.048388 2899 ts_tablet_manager.cc:1397] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250625 08:04:39.048519 2905 tablet_bootstrap.cc:654] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:39.048702 2904 ts_tablet_manager.cc:1905] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:39.048887 2899 raft_consensus.cc:357] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.049003 2899 raft_consensus.cc:383] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.049036 2899 raft_consensus.cc:738] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.050256 2899 consensus_queue.cc:260] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.050891 2904 ts_tablet_manager.cc:1918] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.050954 2904 log.cc:1199] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/d6b8df079002443e9a6b72c15c7c47b7
I20250625 08:04:39.051191 2904 ts_tablet_manager.cc:1939] T d6b8df079002443e9a6b72c15c7c47b7 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:39.051296 2899 ts_tablet_manager.cc:1428] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:39.051770 2904 tablet_replica.cc:331] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:39.051887 2904 raft_consensus.cc:2241] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:39.051967 2899 tablet_bootstrap.cc:492] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:39.051982 2904 raft_consensus.cc:2270] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.052287 2904 ts_tablet_manager.cc:1905] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:39.053167 2899 tablet_bootstrap.cc:654] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:39.054137 2904 ts_tablet_manager.cc:1918] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.054199 2904 log.cc:1199] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/9c3205b0966048aba05895ad53861642
I20250625 08:04:39.054292 2900 tablet_replica.cc:331] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:39.054483 2900 raft_consensus.cc:2241] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:39.054534 2904 ts_tablet_manager.cc:1939] T 9c3205b0966048aba05895ad53861642 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:39.054615 2900 raft_consensus.cc:2270] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:39.055706 2901 tablet_bootstrap.cc:492] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:39.055789 2901 ts_tablet_manager.cc:1397] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20250625 08:04:39.056349 2901 raft_consensus.cc:357] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.056460 2901 raft_consensus.cc:383] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.056492 2901 raft_consensus.cc:738] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.056622 2901 consensus_queue.cc:260] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.056895 2901 ts_tablet_manager.cc:1428] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:39.057060 2901 tablet_bootstrap.cc:492] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:39.057313 2900 ts_tablet_manager.cc:1905] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:39.058153 2901 tablet_bootstrap.cc:654] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:39.059085 2905 tablet_bootstrap.cc:492] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:39.059181 2905 ts_tablet_manager.cc:1397] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20250625 08:04:39.059695 2905 raft_consensus.cc:357] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.059782 2905 raft_consensus.cc:383] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.059803 2905 raft_consensus.cc:738] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.059947 2905 consensus_queue.cc:260] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.060242 2905 ts_tablet_manager.cc:1428] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:39.060286 2899 tablet_bootstrap.cc:492] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:39.060391 2899 ts_tablet_manager.cc:1397] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 08:04:39.060431 2905 tablet_bootstrap.cc:492] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:39.060869 2899 raft_consensus.cc:357] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.060971 2899 raft_consensus.cc:383] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.061003 2899 raft_consensus.cc:738] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.061125 2899 consensus_queue.cc:260] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.061484 2905 tablet_bootstrap.cc:654] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:39.063612 2900 ts_tablet_manager.cc:1918] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:39.063688 2900 log.cc:1199] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/17e0682e2bfe43b7a445f595d8929034
I20250625 08:04:39.063879 2900 ts_tablet_manager.cc:1939] T 17e0682e2bfe43b7a445f595d8929034 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:39.064668 2899 ts_tablet_manager.cc:1428] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 08:04:39.064895 2899 tablet_bootstrap.cc:492] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:39.065268 2731 raft_consensus.cc:491] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:39.065344 2731 raft_consensus.cc:513] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.065744 2731 leader_election.cc:290] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:39.066025 2899 tablet_bootstrap.cc:654] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:39.066288 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97c4e286a5040b1ad425242d9cb2dea" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:39.066418 1982 raft_consensus.cc:2466] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:39.066411 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97c4e286a5040b1ad425242d9cb2dea" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
W20250625 08:04:39.066830 1647 leader_election.cc:343] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:04:39.066951 1647 leader_election.cc:304] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters: de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:39.067116 2731 raft_consensus.cc:2802] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:39.067174 2731 raft_consensus.cc:491] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:39.067214 2731 raft_consensus.cc:3058] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.067279 2901 tablet_bootstrap.cc:492] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:39.067377 2901 ts_tablet_manager.cc:1397] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250625 08:04:39.067838 2901 raft_consensus.cc:357] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.067947 2901 raft_consensus.cc:383] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.067981 2901 raft_consensus.cc:738] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.068109 2901 consensus_queue.cc:260] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.068251 2731 raft_consensus.cc:513] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.068425 2901 ts_tablet_manager.cc:1428] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:39.068601 2731 leader_election.cc:290] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:39.068629 2901 tablet_bootstrap.cc:492] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:39.069144 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97c4e286a5040b1ad425242d9cb2dea" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:39.069749 2901 tablet_bootstrap.cc:654] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
W20250625 08:04:39.070590 1647 leader_election.cc:343] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:04:39.070953 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97c4e286a5040b1ad425242d9cb2dea" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:39.071081 1982 raft_consensus.cc:3058] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.071197 2905 tablet_bootstrap.cc:492] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:39.071287 2905 ts_tablet_manager.cc:1397] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250625 08:04:39.071487 2899 tablet_bootstrap.cc:492] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:39.071565 2899 ts_tablet_manager.cc:1397] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250625 08:04:39.071846 2905 raft_consensus.cc:357] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.071947 2905 raft_consensus.cc:383] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.071979 2905 raft_consensus.cc:738] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.072079 2899 raft_consensus.cc:357] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.072106 2905 consensus_queue.cc:260] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.072116 1982 raft_consensus.cc:2466] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:39.072189 2899 raft_consensus.cc:383] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.072222 2899 raft_consensus.cc:738] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.072338 2899 consensus_queue.cc:260] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.072428 2905 ts_tablet_manager.cc:1428] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:39.072636 2899 ts_tablet_manager.cc:1428] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:39.073096 1647 leader_election.cc:304] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters: de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:39.073274 2731 raft_consensus.cc:2802] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:39.073359 2731 raft_consensus.cc:695] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:39.073500 2731 consensus_queue.cc:237] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.074949 2901 tablet_bootstrap.cc:492] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:39.075044 2901 ts_tablet_manager.cc:1397] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250625 08:04:39.075558 2901 raft_consensus.cc:357] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.075664 2901 raft_consensus.cc:383] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:39.075700 2901 raft_consensus.cc:738] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:39.075834 2901 consensus_queue.cc:260] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.076128 2901 ts_tablet_manager.cc:1428] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 08:04:39.085283 2732 raft_consensus.cc:491] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:39.085425 2732 raft_consensus.cc:513] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.085894 2732 leader_election.cc:290] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:39.086243 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee517261551246b8bd14f8bb2460895c" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:39.086331 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee517261551246b8bd14f8bb2460895c" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:39.086377 1982 raft_consensus.cc:2466] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:39.086673 1848 raft_consensus.cc:2466] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:39.086691 1647 leader_election.cc:304] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:39.086944 2732 raft_consensus.cc:2802] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:39.087000 2732 raft_consensus.cc:491] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:39.087037 2732 raft_consensus.cc:3058] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.088155 2732 raft_consensus.cc:513] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.088547 2732 leader_election.cc:290] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:39.088907 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee517261551246b8bd14f8bb2460895c" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:39.089028 1982 raft_consensus.cc:3058] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.089004 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee517261551246b8bd14f8bb2460895c" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:39.089107 1848 raft_consensus.cc:3058] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.089996 1848 raft_consensus.cc:2466] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:39.089998 1982 raft_consensus.cc:2466] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:39.090308 1647 leader_election.cc:304] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:39.090569 2732 raft_consensus.cc:2802] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:39.090675 2732 raft_consensus.cc:695] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:39.090837 2732 consensus_queue.cc:237] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.140090 2732 raft_consensus.cc:491] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:39.140240 2732 raft_consensus.cc:513] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.140702 2732 leader_election.cc:290] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:39.141260 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fb1337f4e44450bbf1379ecddae8bcf" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:39.141260 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fb1337f4e44450bbf1379ecddae8bcf" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:39.141414 1848 raft_consensus.cc:2466] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:39.141413 1982 raft_consensus.cc:2466] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:39.141726 1647 leader_election.cc:304] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:39.141952 2732 raft_consensus.cc:2802] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:39.142023 2732 raft_consensus.cc:491] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:39.142079 2732 raft_consensus.cc:3058] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.143018 2732 raft_consensus.cc:513] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.143354 2732 leader_election.cc:290] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:39.143814 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fb1337f4e44450bbf1379ecddae8bcf" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:39.143814 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fb1337f4e44450bbf1379ecddae8bcf" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:39.143925 1982 raft_consensus.cc:3058] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.143937 1848 raft_consensus.cc:3058] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.144685 1848 raft_consensus.cc:2466] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:39.144685 1982 raft_consensus.cc:2466] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:39.144981 1647 leader_election.cc:304] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:39.145159 2732 raft_consensus.cc:2802] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:39.145299 2732 raft_consensus.cc:695] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:39.145586 2732 consensus_queue.cc:237] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
W20250625 08:04:39.223963 1782 connection.cc:537] server connection from 127.0.0.1:56124 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:39.223986 1917 connection.cc:537] server connection from 127.0.0.1:35484 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:39.224403 1916 connection.cc:537] client connection to 127.1.19.125:38317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:39.224623 1644 connection.cc:537] client connection to 127.1.19.125:38317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:39.226769 2036 connection.cc:537] client connection to 127.1.19.125:38317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:39.227223 2036 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.19.125:38317)
W20250625 08:04:39.227550 2036 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.19.125:38317)
W20250625 08:04:39.227654 2036 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.125:38317)
W20250625 08:04:39.227855 2036 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.19.125:38317)
W20250625 08:04:39.229375 2870 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:39.226694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.125:38317 (local address 127.1.19.67:48871)
0625 08:04:39.226880 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:39.226893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:39.226974 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:39.229268 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":113}
W20250625 08:04:39.229463 2664 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:39.226471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.125:38317 (local address 127.1.19.65:59267)
0625 08:04:39.226641 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:39.226656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:39.226737 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:39.229355 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":52}
W20250625 08:04:39.229830 2871 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:39.225649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.125:38317 (local address 127.1.19.66:37575)
0625 08:04:39.225889 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:39.225906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:39.226050 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:39.229715 (+ 3665us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":119}
W20250625 08:04:39.230145 2331 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:39.228747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.125:38317 (local address 127.0.0.1:51268)
0625 08:04:39.228903 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:39.228916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:39.228986 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:39.230042 (+ 1056us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":61}
I20250625 08:04:39.253547 2764 raft_consensus.cc:491] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:39.253693 2764 raft_consensus.cc:513] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.254179 2764 leader_election.cc:290] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:39.254676 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42cc323e9e8a4b6f9f58b577c9057d2a" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:39.254781 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42cc323e9e8a4b6f9f58b577c9057d2a" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:39.254830 1711 raft_consensus.cc:2466] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:39.254916 1982 raft_consensus.cc:2466] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:39.255155 1784 leader_election.cc:304] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:39.255369 2764 raft_consensus.cc:2802] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:39.255448 2764 raft_consensus.cc:491] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:39.255481 2764 raft_consensus.cc:3058] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.256319 2764 raft_consensus.cc:513] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.256635 2764 leader_election.cc:290] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:39.257010 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42cc323e9e8a4b6f9f58b577c9057d2a" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:39.257089 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42cc323e9e8a4b6f9f58b577c9057d2a" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:39.257143 1982 raft_consensus.cc:3058] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.257212 1711 raft_consensus.cc:3058] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.258162 1982 raft_consensus.cc:2466] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:39.258162 1711 raft_consensus.cc:2466] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:39.258551 1784 leader_election.cc:304] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:39.258718 2764 raft_consensus.cc:2802] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:39.258886 2764 raft_consensus.cc:695] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:39.259150 2764 consensus_queue.cc:237] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.393262 2764 raft_consensus.cc:491] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:39.393410 2764 raft_consensus.cc:513] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:39.393863 2764 leader_election.cc:290] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:39.394299 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97c4e286a5040b1ad425242d9cb2dea" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:39.394371 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97c4e286a5040b1ad425242d9cb2dea" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:39.394547 1982 raft_consensus.cc:2391] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de62034f80d9463dbd6386fc3e90fdaf in current term 1: Already voted for candidate eebfa8472fe04080ac012e2c91bedaae in this term.
I20250625 08:04:39.394850 1784 leader_election.cc:304] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf; no voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae
I20250625 08:04:39.395051 2764 raft_consensus.cc:3058] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:39.395941 2764 raft_consensus.cc:2747] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 08:04:39.477548 2732 consensus_queue.cc:1035] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:39.481150 2661 consensus_queue.cc:1035] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:39.504281 2941 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:39.504395 2941 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:39.504808 2941 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:39.505828 2376 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:39.505972 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:39.506055 2375 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:39.506186 2941 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250625 08:04:39.530555 2661 consensus_queue.cc:1035] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:39.534569 2732 consensus_queue.cc:1035] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:39.620723 2661 consensus_queue.cc:1035] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:39.626283 2732 consensus_queue.cc:1035] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:39.708130 2764 consensus_queue.cc:1035] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:39.711625 2930 consensus_queue.cc:1035] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:40.240286 2955 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:40.240420 2955 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:40.240866 2955 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:40.242137 2376 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:40.242223 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:40.242316 2375 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:40.242499 2955 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250625 08:04:40.263880 2732 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:40.264024 2732 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:40.264449 2732 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:04:40.265311 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:04:40.265476 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:04:40.265529 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:40.265697 2732 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:40.711022 2731 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:40.711165 2731 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:40.711583 2731 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:04:40.712329 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:04:40.712477 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:40.712536 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:40.712704 2731 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:04:40.784373 1647 consensus_peers.cc:487] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7342ad43df3d493b994075493c6ab8a4. This is attempt 6: this message will repeat every 5th retry.
I20250625 08:04:40.842561 2966 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:40.842692 2966 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:40.843171 2966 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:40.844206 2376 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:40.844296 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:40.844355 2375 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:40.844488 2966 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250625 08:04:40.909032 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 6: this message will repeat every 5th retry.
W20250625 08:04:40.964135 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 6: this message will repeat every 5th retry.
W20250625 08:04:41.054795 1647 consensus_peers.cc:487] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7342ad43df3d493b994075493c6ab8a4. This is attempt 6: this message will repeat every 5th retry.
I20250625 08:04:41.613056 2360 master_runner.cc:186] Error getting master registration for 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111),
I20250625 08:04:41.613199 2360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:41.714780 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.124:37745
--webserver_interface=127.1.19.124
--webserver_port=44175
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 08:04:41.831272 2977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:41.831619 2977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:41.831679 2977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:41.835469 2977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:41.835551 2977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:41.835573 2977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:41.835592 2977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:41.840016 2977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.124:37745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.19.124
--webserver_port=44175
--never_fsync=true
--heap_profile_path=/tmp/kudu.2977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:41.841195 2977 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:41.842355 2977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:41.849596 2987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:41.849596 2989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:41.849718 2977 server_base.cc:1048] running on GCE node
W20250625 08:04:41.849596 2986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:41.850517 2977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:41.851609 2977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:41.852811 2977 hybrid_clock.cc:648] HybridClock initialized: now 1750838681852788 us; error 45 us; skew 500 ppm
I20250625 08:04:41.855099 2977 webserver.cc:469] Webserver started at http://127.1.19.124:44175/ using document root <none> and password file <none>
I20250625 08:04:41.855762 2977 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:41.855831 2977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:41.859707 2977 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250625 08:04:41.861977 2995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:41.863193 2977 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250625 08:04:41.863335 2977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
uuid: "755093bec4b54a9588598f814c3708e1"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:41.863785 2977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:41.882315 2977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:41.883085 2977 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:41.883271 2977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:41.891301 2977 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.124:37745
I20250625 08:04:41.891346 3047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.124:37745 every 8 connection(s)
I20250625 08:04:41.892632 2977 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
I20250625 08:04:41.896746 3048 sys_catalog.cc:263] Verifying existing consensus state
I20250625 08:04:41.898757 3048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap starting.
I20250625 08:04:41.902758 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 2977
I20250625 08:04:41.909096 3048 log.cc:826] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:41.941622 3048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=34 ignored=0} mutations{seen=34 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:41.942083 3048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap complete.
I20250625 08:04:41.944893 3048 raft_consensus.cc:357] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:41.945274 3048 raft_consensus.cc:738] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755093bec4b54a9588598f814c3708e1, State: Initialized, Role: FOLLOWER
I20250625 08:04:41.945731 3048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 4.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:41.946213 3052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:41.946354 3052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:41.946965 3048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:41.954558 3066 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:41.954741 3066 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: loading cluster ID for follower catalog manager: success
I20250625 08:04:41.957743 3066 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: acquiring CA information for follower catalog manager: success
I20250625 08:04:41.959996 3066 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:41.960898 2977 master_runner.cc:186] Error getting master registration for 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111),
I20250625 08:04:41.963021 2977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:41.974704 3071 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:41.974942 3071 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:41.976262 3071 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:41.978029 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:41.981585 3022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 8 candidate_status { last_received { term: 7 index: 73 } } ignore_live_leader: false dest_uuid: "755093bec4b54a9588598f814c3708e1" is_pre_election: true
I20250625 08:04:41.981981 3022 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f0b114bd68344b0bd2ee322bd5eccdc in term 4.
I20250625 08:04:41.982515 2376 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22
I20250625 08:04:41.983801 3071 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250625 08:04:41.983927 3071 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0404561252194ce29bf2924fd5e67e22)
I20250625 08:04:41.984038 3071 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 7 FOLLOWER]: Advancing to term 8
I20250625 08:04:41.985298 3071 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
W20250625 08:04:41.986850 2375 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:41.987143 3071 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 election: Requested vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:41.988590 3022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 8 candidate_status { last_received { term: 7 index: 73 } } ignore_live_leader: false dest_uuid: "755093bec4b54a9588598f814c3708e1"
I20250625 08:04:41.988754 3022 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 4 FOLLOWER]: Advancing to term 8
I20250625 08:04:41.989512 1757 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:41.991384 3022 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f0b114bd68344b0bd2ee322bd5eccdc in term 8.
I20250625 08:04:41.991729 2376 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22
I20250625 08:04:41.991978 3071 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 8 FOLLOWER]: Leader election won for term 8
I20250625 08:04:41.992182 3012 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" instance_seqno: 1750838666815429) as {username='slave'} at 127.1.19.65:39283; Asking this server to re-register.
I20250625 08:04:41.992327 3071 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 8 LEADER]: Becoming Leader. State: Replica: 5f0b114bd68344b0bd2ee322bd5eccdc, State: Running, Role: LEADER
I20250625 08:04:41.992686 3071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 7.73, Last appended by leader: 73, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:41.992791 1757 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:41.993791 3071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0b114bd68344b0bd2ee322bd5eccdc. Latest consensus state: current_term: 8 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:41.993974 3071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:41.994031 3012 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:41.994328 3076 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:41.997648 3076 catalog_manager.cc:671] Loaded metadata for table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]
I20250625 08:04:41.998056 3076 catalog_manager.cc:671] Loaded metadata for table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]
I20250625 08:04:41.998353 3076 catalog_manager.cc:671] Loaded metadata for table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]
I20250625 08:04:41.998660 3076 catalog_manager.cc:671] Loaded metadata for table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]
I20250625 08:04:41.998889 3076 catalog_manager.cc:671] Loaded metadata for table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]
I20250625 08:04:41.999107 3076 catalog_manager.cc:671] Loaded metadata for table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]
I20250625 08:04:41.999327 3076 catalog_manager.cc:671] Loaded metadata for table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]
I20250625 08:04:41.999565 3076 catalog_manager.cc:671] Loaded metadata for table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]
I20250625 08:04:42.001307 3076 tablet_loader.cc:96] loaded metadata for tablet 1564b8dab2e9429f9c2a98f0935928f5 (table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d])
I20250625 08:04:42.001866 3076 tablet_loader.cc:96] loaded metadata for tablet 2fb1337f4e44450bbf1379ecddae8bcf (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875])
I20250625 08:04:42.002270 3076 tablet_loader.cc:96] loaded metadata for tablet 42cc323e9e8a4b6f9f58b577c9057d2a (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a])
I20250625 08:04:42.002475 3076 tablet_loader.cc:96] loaded metadata for tablet 48f3f72f9ca843298509d81fb5c4232e (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020])
I20250625 08:04:42.003562 3076 tablet_loader.cc:96] loaded metadata for tablet b97c4e286a5040b1ad425242d9cb2dea (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6])
I20250625 08:04:42.004397 3076 tablet_loader.cc:96] loaded metadata for tablet ddb06adb0b7d4f888e489eb5721321fc (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089])
I20250625 08:04:42.004594 3076 tablet_loader.cc:96] loaded metadata for tablet ee4801582108472c922fea326dbb21cf (table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904])
I20250625 08:04:42.004776 3076 tablet_loader.cc:96] loaded metadata for tablet ee517261551246b8bd14f8bb2460895c (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206])
I20250625 08:04:42.005033 3076 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:42.005915 3076 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:42.005976 3076 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:42.007017 3076 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:42.007781 3076 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Loaded TSK: 0
I20250625 08:04:42.008735 3076 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:42.009492 3076 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:42.010231 3076 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 20
I20250625 08:04:42.046473 2391 master_service.cc:948] ReplaceTablet: received request to replace tablet 1564b8dab2e9429f9c2a98f0935928f5 from {username='slave'} at 127.0.0.1:46090
I20250625 08:04:42.047065 2384 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46090:
name: "default.Table_035cffaa28ec4a838bb218bb37230370"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:42.047580 2390 master_service.cc:948] ReplaceTablet: received request to replace tablet 48f3f72f9ca843298509d81fb5c4232e from {username='slave'} at 127.0.0.1:46090
I20250625 08:04:42.047611 2389 master_service.cc:948] ReplaceTablet: received request to replace tablet 48f3f72f9ca843298509d81fb5c4232e from {username='slave'} at 127.0.0.1:46090
I20250625 08:04:42.047766 2388 master_service.cc:948] ReplaceTablet: received request to replace tablet ee517261551246b8bd14f8bb2460895c from {username='slave'} at 127.0.0.1:46090
I20250625 08:04:42.048220 2385 master_service.cc:948] ReplaceTablet: received request to replace tablet ee4801582108472c922fea326dbb21cf from {username='slave'} at 127.0.0.1:46090
I20250625 08:04:42.048512 2387 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46090:
table { table_name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2" } modify_external_catalogs: true
I20250625 08:04:42.048664 2387 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46090:
table { table_name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2" } modify_external_catalogs: true
I20250625 08:04:42.049010 2386 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46090:
name: "default.Table_128075ffc1db4314b52b7c28536f6e89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:42.049006 2383 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46090:
name: "default.Table_174089cfe43b4dbb9ed1b9c2921f9187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:42.050812 3022 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Refusing update from remote peer 5f0b114bd68344b0bd2ee322bd5eccdc: Log matching property violated. Preceding OpId in replica: term: 4 index: 53. Preceding OpId from leader: term: 8 index: 75. (index mismatch)
I20250625 08:04:42.049052 2382 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46090:
table { table_name: "default.Table_23c54834c68a47bd9654b34281b53e7d" } new_table_name: "default.Table_fbd9fab6fc2a480f88e12bb1ada0b9f6" modify_external_catalogs: true
I20250625 08:04:42.051491 3071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 53, Time since last communication: 0.000s
W20250625 08:04:42.051892 2375 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc -> Peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Couldn't send request to peer 0404561252194ce29bf2924fd5e67e22. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:42.055518 1686 tablet_service.cc:1515] Processing DeleteTablet for tablet 7342ad43df3d493b994075493c6ab8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet ee4801582108472c922fea326dbb21cf) from {username='slave'} at 127.0.0.1:60736
F20250625 08:04:42.055894 2445 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 4 out of 11 received events: NotificationEvent(eventId=21, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_31023e55a4384ac88f8415871e52d142, message={"server":"","servicePrincipal":"","db":"default","table":"table_31023e55a4384ac88f8415871e52d142","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_31023e55a4384ac88f8415871e52d142\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_31023e55a4384ac88f8415871e52d142\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_31023e55a4384ac88f8415871e52d142\",\"kudu.table_id\":\"7825ea67af2d42b68200097030b08116\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1750838669, eventType=ALTER_TABLE, dbName=default, tableName=table_7cb71508c2934e1aa091c71056ae7a01, message={"server":"","servicePrincipal":"","db":"default","table":"table_b49cbf7ee55042b39d7b5d3946a35618","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_b49cbf7ee55042b39d7b5d3946a35618\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_b49cbf7ee55042b39d7b5d3946a35618\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b49cbf7ee55042b39d7b5d3946a35618\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7cb71508c2934e1aa091c71056ae7a01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_7cb71508c2934e1aa091c71056ae7a01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7cb71508c2934e1aa091c71056ae7a01\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_d78071777f3646548f7f75cea674ecdc, message={"server":"","servicePrincipal":"","db":"default","table":"table_d78071777f3646548f7f75cea674ecdc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d78071777f3646548f7f75cea674ecdc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_d78071777f3646548f7f75cea674ecdc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d78071777f3646548f7f75cea674ecdc\",\"kudu.table_id\":\"0e43d85f40e545d9b31c5cb3e62f89a8\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=CREATE_TABLE, dbName=default, tableName=table_88079d64160a4318bf8ea1807cb3a30d, message={"server":"","servicePrincipal":"","db":"default","table":"table_88079d64160a4318bf8ea1807cb3a30d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_88079d64160a4318bf8ea1807cb3a30d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_88079d64160a4318bf8ea1807cb3a30d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_88079d64160a4318bf8ea1807cb3a30d\",\"kudu.table_id\":\"7c7e3cbdaf064697a864873ed87a21b6\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250625 08:04:42.057365 3080 tablet_replica.cc:331] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:42.057526 3080 raft_consensus.cc:2241] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:42.057739 3080 raft_consensus.cc:2270] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.058287 3080 ts_tablet_manager.cc:1905] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fb753431dcd google::LogMessage::Fail() at ??:0
I20250625 08:04:42.060774 3080 ts_tablet_manager.cc:1918] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.060848 3080 log.cc:1199] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/7342ad43df3d493b994075493c6ab8a4
I20250625 08:04:42.061131 3080 ts_tablet_manager.cc:1939] T 7342ad43df3d493b994075493c6ab8a4 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:42.061199 3052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0b114bd68344b0bd2ee322bd5eccdc. Latest consensus state: current_term: 8 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:42.061369 3052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:42.061487 3052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
@ 0x7fb753435b93 google::LogMessage::SendToLog() at ??:0
I20250625 08:04:42.061576 3052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:42.061658 3052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:42.061730 3052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:42.061797 3052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:42.061870 3052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:42.062026 2378 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 7342ad43df3d493b994075493c6ab8a4 (table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]) successfully deleted
@ 0x7fb7534317cc google::LogMessage::Flush() at ??:0
@ 0x7fb753431a11 google::LogMessage::~LogMessage() at ??:0
I20250625 08:04:42.066062 3074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:42.066231 3074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:42.069005 3086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:42.069175 3086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:42.072975 3079 mvcc.cc:204] Tried to move back new op lower bound from 7171435241677721600 to 7171435199568863232. Current Snapshot: MvccSnapshot[applied={T|T < 7171435241671499776}]
@ 0x7fb7583cac91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 08:04:42.074092 2385 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Sending DeleteTablet for 3 replicas of tablet ee4801582108472c922fea326dbb21cf
I20250625 08:04:42.074191 2388 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Sending DeleteTablet for 3 replicas of tablet ee517261551246b8bd14f8bb2460895c
I20250625 08:04:42.074793 2385 catalog_manager.cc:6652] ReplaceTablet: tablet ee4801582108472c922fea326dbb21cf deleted and replaced by tablet 90154412b7b04b6db02f912a53000edb
I20250625 08:04:42.074821 2388 catalog_manager.cc:6652] ReplaceTablet: tablet ee517261551246b8bd14f8bb2460895c deleted and replaced by tablet 782a60aff92a4636bc98813f2214d1af
I20250625 08:04:42.074896 2390 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Sending DeleteTablet for 3 replicas of tablet 48f3f72f9ca843298509d81fb5c4232e
I20250625 08:04:42.075135 2385 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46090:
table { table_name: "default.Table_88079d64160a4318bf8ea1807cb3a30d" } modify_external_catalogs: true
I20250625 08:04:42.075266 2385 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46090:
table { table_name: "default.Table_88079d64160a4318bf8ea1807cb3a30d" } modify_external_catalogs: true
I20250625 08:04:42.075510 2389 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46090:
name: "default.Table_f22b18eed5c34ffaa7af847a3c64598d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:42.076409 2391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Sending DeleteTablet for 3 replicas of tablet 1564b8dab2e9429f9c2a98f0935928f5
I20250625 08:04:42.076679 2391 catalog_manager.cc:6652] ReplaceTablet: tablet 1564b8dab2e9429f9c2a98f0935928f5 deleted and replaced by tablet a0db2bca14734321aa2167a9aade37e0
I20250625 08:04:42.076841 2391 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46090:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:42.076962 2391 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46090:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:42.077107 1686 tablet_service.cc:1515] Processing DeleteTablet for tablet ee4801582108472c922fea326dbb21cf with delete_type TABLET_DATA_DELETED (replaced by tablet 90154412b7b04b6db02f912a53000edb) from {username='slave'} at 127.0.0.1:60736
I20250625 08:04:42.078541 1685 tablet_service.cc:1515] Processing DeleteTablet for tablet ee517261551246b8bd14f8bb2460895c with delete_type TABLET_DATA_DELETED (replaced by tablet 782a60aff92a4636bc98813f2214d1af) from {username='slave'} at 127.0.0.1:60736
I20250625 08:04:42.078670 1687 tablet_service.cc:1515] Processing DeleteTablet for tablet 48f3f72f9ca843298509d81fb5c4232e with delete_type TABLET_DATA_DELETED (replaced by tablet 0561d480e8b64f749c705741c3189f7f) from {username='slave'} at 127.0.0.1:60736
I20250625 08:04:42.078809 3080 tablet_replica.cc:331] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
@ 0x7fb7583c9ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 08:04:42.078931 3080 raft_consensus.cc:2241] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:42.079052 3080 raft_consensus.cc:2270] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.079465 3080 ts_tablet_manager.cc:1905] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.081777 3080 ts_tablet_manager.cc:1918] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.081847 3080 log.cc:1199] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/ee4801582108472c922fea326dbb21cf
@ 0x7fb7583c930b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 08:04:42.082105 3080 ts_tablet_manager.cc:1939] T ee4801582108472c922fea326dbb21cf P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:42.075142 2390 catalog_manager.cc:6652] ReplaceTablet: tablet 48f3f72f9ca843298509d81fb5c4232e deleted and replaced by tablet 0561d480e8b64f749c705741c3189f7f
I20250625 08:04:42.083019 2390 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46090:
table { table_name: "default.Table_5eebaa7c05694491963fce48525e4681" } new_table_name: "default.Table_69a1c4361ab540a48fd24a5198251c2c" modify_external_catalogs: true
I20250625 08:04:42.083803 1687 tablet_service.cc:1515] Processing DeleteTablet for tablet 1564b8dab2e9429f9c2a98f0935928f5 with delete_type TABLET_DATA_DELETED (replaced by tablet a0db2bca14734321aa2167a9aade37e0) from {username='slave'} at 127.0.0.1:60736
I20250625 08:04:42.084144 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet ee4801582108472c922fea326dbb21cf with delete_type TABLET_DATA_DELETED (replaced by tablet 90154412b7b04b6db02f912a53000edb) from {username='slave'} at 127.0.0.1:56372
I20250625 08:04:42.084187 1960 tablet_service.cc:1515] Processing DeleteTablet for tablet 48f3f72f9ca843298509d81fb5c4232e with delete_type TABLET_DATA_DELETED (replaced by tablet 0561d480e8b64f749c705741c3189f7f) from {username='slave'} at 127.0.0.1:56372
I20250625 08:04:42.084241 1963 tablet_service.cc:1515] Processing DeleteTablet for tablet 1564b8dab2e9429f9c2a98f0935928f5 with delete_type TABLET_DATA_DELETED (replaced by tablet a0db2bca14734321aa2167a9aade37e0) from {username='slave'} at 127.0.0.1:56372
I20250625 08:04:42.076855 2388 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46090:
table { table_name: "default.Table_a3a8c500599e482da5b7120ec26a797a" } new_table_name: "default.Table_6894ca710ded4040ba337309d454d090" modify_external_catalogs: true
I20250625 08:04:42.084144 1958 tablet_service.cc:1515] Processing DeleteTablet for tablet ee517261551246b8bd14f8bb2460895c with delete_type TABLET_DATA_DELETED (replaced by tablet 782a60aff92a4636bc98813f2214d1af) from {username='slave'} at 127.0.0.1:56372
@ 0x7fb7583cea12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5558e7e69752 std::function<>::operator()() at ??:0
I20250625 08:04:42.087697 1823 tablet_service.cc:1515] Processing DeleteTablet for tablet 48f3f72f9ca843298509d81fb5c4232e with delete_type TABLET_DATA_DELETED (replaced by tablet 0561d480e8b64f749c705741c3189f7f) from {username='slave'} at 127.0.0.1:53284
I20250625 08:04:42.087749 1824 tablet_service.cc:1515] Processing DeleteTablet for tablet ee4801582108472c922fea326dbb21cf with delete_type TABLET_DATA_DELETED (replaced by tablet 90154412b7b04b6db02f912a53000edb) from {username='slave'} at 127.0.0.1:53284
I20250625 08:04:42.087901 1825 tablet_service.cc:1515] Processing DeleteTablet for tablet ee517261551246b8bd14f8bb2460895c with delete_type TABLET_DATA_DELETED (replaced by tablet 782a60aff92a4636bc98813f2214d1af) from {username='slave'} at 127.0.0.1:53284
I20250625 08:04:42.088011 1827 tablet_service.cc:1515] Processing DeleteTablet for tablet 1564b8dab2e9429f9c2a98f0935928f5 with delete_type TABLET_DATA_DELETED (replaced by tablet a0db2bca14734321aa2167a9aade37e0) from {username='slave'} at 127.0.0.1:53284
I20250625 08:04:42.088586 3080 tablet_replica.cc:331] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:42.088752 3080 raft_consensus.cc:2241] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:42.089160 2378 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet ee4801582108472c922fea326dbb21cf (table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]) successfully deleted
I20250625 08:04:42.088999 3080 raft_consensus.cc:2270] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fb7548830b7 kudu::Thread::SuperviseThread() at ??:0
I20250625 08:04:42.089819 3080 ts_tablet_manager.cc:1905] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fb7550726db start_thread at ??:0
I20250625 08:04:42.092908 3080 ts_tablet_manager.cc:1918] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.092994 3080 log.cc:1199] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/ee517261551246b8bd14f8bb2460895c
I20250625 08:04:42.093335 3080 ts_tablet_manager.cc:1939] T ee517261551246b8bd14f8bb2460895c P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
@ 0x7fb7525e871f clone at ??:0
I20250625 08:04:42.094074 3080 tablet_replica.cc:331] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:42.094228 3080 raft_consensus.cc:2241] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:42.095000 3097 tablet_replica.cc:331] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:42.095147 1958 tablet_service.cc:1468] Processing CreateTablet for tablet 782a60aff92a4636bc98813f2214d1af (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:42.095340 3098 tablet_replica.cc:331] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:42.095489 3098 raft_consensus.cc:2241] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:42.095522 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 782a60aff92a4636bc98813f2214d1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:42.095621 3098 raft_consensus.cc:2270] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.096082 3098 ts_tablet_manager.cc:1905] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.094671 1687 tablet_service.cc:1468] Processing CreateTablet for tablet 782a60aff92a4636bc98813f2214d1af (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:42.097709 1687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 782a60aff92a4636bc98813f2214d1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:42.095163 3097 raft_consensus.cc:2241] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:42.098192 3097 raft_consensus.cc:2270] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.098636 3098 ts_tablet_manager.cc:1918] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.098655 3097 ts_tablet_manager.cc:1905] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.098716 3098 log.cc:1199] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/48f3f72f9ca843298509d81fb5c4232e
I20250625 08:04:42.098984 3098 ts_tablet_manager.cc:1939] T 48f3f72f9ca843298509d81fb5c4232e P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:42.099627 3098 tablet_replica.cc:331] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:42.099764 3098 raft_consensus.cc:2241] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:42.099865 3098 raft_consensus.cc:2270] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.100229 3098 ts_tablet_manager.cc:1905] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.100734 3097 ts_tablet_manager.cc:1918] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.100770 3080 raft_consensus.cc:2270] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.100800 3097 log.cc:1199] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/ee4801582108472c922fea326dbb21cf
I20250625 08:04:42.101071 3097 ts_tablet_manager.cc:1939] T ee4801582108472c922fea326dbb21cf P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:42.101146 3080 ts_tablet_manager.cc:1905] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.101337 3099 tablet_bootstrap.cc:492] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:42.101805 3097 tablet_replica.cc:331] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:42.101917 3097 raft_consensus.cc:2241] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:42.102128 3097 raft_consensus.cc:2270] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.102155 3098 ts_tablet_manager.cc:1918] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.102212 3098 log.cc:1199] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/ee4801582108472c922fea326dbb21cf
I20250625 08:04:42.102499 3098 ts_tablet_manager.cc:1939] T ee4801582108472c922fea326dbb21cf P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:42.102573 3097 ts_tablet_manager.cc:1905] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.102653 3099 tablet_bootstrap.cc:654] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:42.103082 3098 tablet_replica.cc:331] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:42.103190 3098 raft_consensus.cc:2241] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:42.103284 3098 raft_consensus.cc:2270] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.103597 3098 ts_tablet_manager.cc:1905] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.103720 3099 tablet_bootstrap.cc:492] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:42.103796 3099 ts_tablet_manager.cc:1397] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:42.104221 3099 raft_consensus.cc:357] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:42.104321 3099 raft_consensus.cc:383] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:42.104350 3099 raft_consensus.cc:738] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:42.104471 3099 consensus_queue.cc:260] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:42.104723 3099 ts_tablet_manager.cc:1428] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:42.104825 3097 ts_tablet_manager.cc:1918] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.104889 3097 log.cc:1199] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/48f3f72f9ca843298509d81fb5c4232e
I20250625 08:04:42.105139 3097 ts_tablet_manager.cc:1939] T 48f3f72f9ca843298509d81fb5c4232e P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:42.105722 3098 ts_tablet_manager.cc:1918] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.105793 3098 log.cc:1199] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/ee517261551246b8bd14f8bb2460895c
I20250625 08:04:42.106014 3098 ts_tablet_manager.cc:1939] T ee517261551246b8bd14f8bb2460895c P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:42.106788 3098 tablet_replica.cc:331] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:42.106909 3098 raft_consensus.cc:2241] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:42.107004 3098 raft_consensus.cc:2270] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.107321 3097 tablet_replica.cc:331] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:42.107394 3098 ts_tablet_manager.cc:1905] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.107434 3097 raft_consensus.cc:2241] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:42.107631 3097 raft_consensus.cc:2270] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.107967 3097 ts_tablet_manager.cc:1905] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.109258 3098 ts_tablet_manager.cc:1918] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.109323 3098 log.cc:1199] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/1564b8dab2e9429f9c2a98f0935928f5
I20250625 08:04:42.109562 3098 ts_tablet_manager.cc:1939] T 1564b8dab2e9429f9c2a98f0935928f5 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:42.110289 3097 ts_tablet_manager.cc:1918] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.110349 3097 log.cc:1199] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/1564b8dab2e9429f9c2a98f0935928f5
I20250625 08:04:42.110626 3097 ts_tablet_manager.cc:1939] T 1564b8dab2e9429f9c2a98f0935928f5 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:42.111095 3080 ts_tablet_manager.cc:1918] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.111167 3080 log.cc:1199] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/48f3f72f9ca843298509d81fb5c4232e
I20250625 08:04:42.111410 3080 ts_tablet_manager.cc:1939] T 48f3f72f9ca843298509d81fb5c4232e P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:42.111497 2029 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:42.111967 3097 tablet_replica.cc:331] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:42.111970 3080 tablet_replica.cc:331] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:42.112080 3097 raft_consensus.cc:2241] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:42.112080 3080 raft_consensus.cc:2241] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:42.112179 3097 raft_consensus.cc:2270] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.112179 3080 raft_consensus.cc:2270] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:42.112526 3097 ts_tablet_manager.cc:1905] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.112526 3080 ts_tablet_manager.cc:1905] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:42.112829 3012 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" instance_seqno: 1750838667205184) as {username='slave'} at 127.1.19.67:42543; Asking this server to re-register.
I20250625 08:04:42.113278 2029 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:42.114137 3012 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:42.114413 3097 ts_tablet_manager.cc:1918] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.114517 3097 log.cc:1199] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/ee517261551246b8bd14f8bb2460895c
I20250625 08:04:42.114758 3097 ts_tablet_manager.cc:1939] T ee517261551246b8bd14f8bb2460895c P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:42.115203 3100 tablet_bootstrap.cc:492] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:42.116324 3100 tablet_bootstrap.cc:654] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:42.114413 3080 ts_tablet_manager.cc:1918] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:42.116803 3080 log.cc:1199] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/1564b8dab2e9429f9c2a98f0935928f5
I20250625 08:04:42.117033 3080 ts_tablet_manager.cc:1939] T 1564b8dab2e9429f9c2a98f0935928f5 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:42.117812 3100 tablet_bootstrap.cc:492] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:42.117873 3100 ts_tablet_manager.cc:1397] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:42.118220 3100 raft_consensus.cc:357] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:42.118323 3100 raft_consensus.cc:383] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:42.118355 3100 raft_consensus.cc:738] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:42.118572 3100 consensus_queue.cc:260] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:42.118858 1758 heartbeater.cc:499] Master 127.1.19.126:36875 was elected leader, sending a full tablet report...
I20250625 08:04:42.118858 3100 ts_tablet_manager.cc:1428] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 08:04:42.164793 2766 raft_consensus.cc:491] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:42.164942 2766 raft_consensus.cc:513] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:42.165395 2766 leader_election.cc:290] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:42.165926 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "782a60aff92a4636bc98813f2214d1af" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:42.166065 1711 raft_consensus.cc:2466] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
W20250625 08:04:42.166270 1919 leader_election.cc:343] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 782a60aff92a4636bc98813f2214d1af
I20250625 08:04:42.166622 1919 leader_election.cc:304] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters: de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:42.166919 2766 raft_consensus.cc:2802] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:42.166981 2766 raft_consensus.cc:491] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:42.167021 2766 raft_consensus.cc:3058] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:42.168201 2766 raft_consensus.cc:513] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:42.168599 2766 leader_election.cc:290] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:42.169198 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "782a60aff92a4636bc98813f2214d1af" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:42.169332 1711 raft_consensus.cc:3058] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
W20250625 08:04:42.169461 1919 leader_election.cc:343] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 782a60aff92a4636bc98813f2214d1af
I20250625 08:04:42.170367 1711 raft_consensus.cc:2466] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:42.170776 1919 leader_election.cc:304] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters: de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:42.171018 2766 raft_consensus.cc:2802] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:42.171233 2766 raft_consensus.cc:695] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:42.171401 2766 consensus_queue.cc:237] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:42.297348 1894 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:42.298666 3012 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" instance_seqno: 1750838667017582) as {username='slave'} at 127.1.19.66:48307; Asking this server to re-register.
I20250625 08:04:42.299098 1894 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:42.299940 3012 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:04:42.301872 1784 connection.cc:537] server connection from 127.0.0.1:53284 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:42.301872 1919 connection.cc:537] server connection from 127.0.0.1:56372 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:42.301927 1647 connection.cc:537] server connection from 127.0.0.1:60736 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:42.302150 1647 connection.cc:537] client connection to 127.1.19.126:36875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:42.302181 1919 connection.cc:537] client connection to 127.1.19.126:36875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:42.304224 1784 connection.cc:537] client connection to 127.1.19.126:36875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:42.304275 2039 connection.cc:537] client connection to 127.1.19.126:36875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:42.304448 2978 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:42.302864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.126:36875 (local address 127.1.19.67:51307)
0625 08:04:42.303032 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:42.303052 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:42.303151 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:42.304314 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":67}
W20250625 08:04:42.304875 2039 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.19.126:36875)
W20250625 08:04:42.305020 2933 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:42.302887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.126:36875 (local address 127.1.19.65:36731)
0625 08:04:42.303056 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:42.303069 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:42.303156 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:42.304902 (+ 1746us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":96}
W20250625 08:04:42.305225 2039 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.126:36875)
W20250625 08:04:42.305290 2039 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.19.126:36875)
W20250625 08:04:42.305569 2039 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.19.126:36875)
W20250625 08:04:42.569494 1919 consensus_peers.cc:487] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 782a60aff92a4636bc98813f2214d1af. This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:42.591256 3120 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:42.591389 3120 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:42.592741 3120 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
W20250625 08:04:42.593329 2999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:04:42.594106 2996 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:42.594251 2999 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:04:42.594360 2999 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc
I20250625 08:04:42.594655 3120 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 08:04:42.670918 2661 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:42.671072 2661 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:42.671535 2661 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:04:42.672331 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:04:42.672477 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:42.672550 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:42.672748 2661 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:42.711138 3113 consensus_queue.cc:1035] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:43.178498 3129 ts_tablet_manager.cc:927] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: Initiating tablet copy from peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:43.179097 3129 tablet_copy_client.cc:323] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: tablet copy: Beginning tablet copy session from remote peer at address 127.1.19.67:37005
I20250625 08:04:43.179697 2003 tablet_copy_service.cc:140] P 5dae8595b9484b57bc7ccea543f83130: Received BeginTabletCopySession request for tablet 782a60aff92a4636bc98813f2214d1af from peer de62034f80d9463dbd6386fc3e90fdaf ({username='slave'} at 127.1.19.66:41797)
I20250625 08:04:43.179809 2003 tablet_copy_service.cc:161] P 5dae8595b9484b57bc7ccea543f83130: Beginning new tablet copy session on tablet 782a60aff92a4636bc98813f2214d1af from peer de62034f80d9463dbd6386fc3e90fdaf at {username='slave'} at 127.1.19.66:41797: session id = de62034f80d9463dbd6386fc3e90fdaf-782a60aff92a4636bc98813f2214d1af
I20250625 08:04:43.180572 2003 tablet_copy_source_session.cc:215] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 08:04:43.181772 3129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 782a60aff92a4636bc98813f2214d1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:43.183840 3129 tablet_copy_client.cc:806] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: tablet copy: Starting download of 0 data blocks...
I20250625 08:04:43.184024 3129 tablet_copy_client.cc:670] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: tablet copy: Starting download of 1 WAL segments...
I20250625 08:04:43.185410 3129 tablet_copy_client.cc:538] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 08:04:43.186822 3129 tablet_bootstrap.cc:492] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:43.190336 3129 tablet_bootstrap.cc:492] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:43.190626 3129 tablet_bootstrap.cc:492] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap complete.
I20250625 08:04:43.190805 3129 ts_tablet_manager.cc:1397] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250625 08:04:43.191262 3129 raft_consensus.cc:357] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:43.191363 3129 raft_consensus.cc:738] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:43.191509 3129 consensus_queue.cc:260] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:43.191962 3129 ts_tablet_manager.cc:1428] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:43.192906 2003 tablet_copy_service.cc:342] P 5dae8595b9484b57bc7ccea543f83130: Request end of tablet copy session de62034f80d9463dbd6386fc3e90fdaf-782a60aff92a4636bc98813f2214d1af received from {username='slave'} at 127.1.19.66:41797
I20250625 08:04:43.193006 2003 tablet_copy_service.cc:434] P 5dae8595b9484b57bc7ccea543f83130: ending tablet copy session de62034f80d9463dbd6386fc3e90fdaf-782a60aff92a4636bc98813f2214d1af on tablet 782a60aff92a4636bc98813f2214d1af with peer de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:43.196727 3133 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:43.196843 3133 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:43.197225 3133 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
W20250625 08:04:43.198341 2999 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:04:43.198423 2996 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:43.198540 2996 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc
I20250625 08:04:43.198680 3133 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 08:04:43.633921 1847 raft_consensus.cc:1215] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250625 08:04:43.822928 3142 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:43.823062 3142 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:43.823518 3142 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
W20250625 08:04:43.824777 2996 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:43.824874 2999 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:04:43.824959 2999 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc
I20250625 08:04:43.825098 3142 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 08:04:44.356478 3151 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:44.356631 3151 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:44.357093 3151 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
W20250625 08:04:44.358525 2996 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:44.358621 2999 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:04:44.358683 2999 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc
I20250625 08:04:44.358834 3151 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250625 08:04:44.650123 2732 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:44.650276 2732 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:44.650739 2732 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:04:44.651499 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:04:44.651635 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:04:44.651692 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:44.651844 2732 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:45.096120 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.126:36875
--webserver_interface=127.1.19.126
--webserver_port=32959
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 08:04:45.208014 3160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:45.208250 3160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:45.208298 3160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:45.211717 3160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:45.211773 3160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:45.211794 3160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:45.211812 3160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:45.215968 3160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.126:36875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.19.126
--webserver_port=32959
--never_fsync=true
--heap_profile_path=/tmp/kudu.3160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:45.217103 3160 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:45.218235 3160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:45.225209 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:45.225140 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:45.225276 3160 server_base.cc:1048] running on GCE node
W20250625 08:04:45.225147 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:45.225930 3160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:45.226950 3160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:45.228113 3160 hybrid_clock.cc:648] HybridClock initialized: now 1750838685228103 us; error 27 us; skew 500 ppm
I20250625 08:04:45.236076 3160 webserver.cc:469] Webserver started at http://127.1.19.126:32959/ using document root <none> and password file <none>
I20250625 08:04:45.236696 3160 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:45.236756 3160 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:45.240329 3160 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250625 08:04:45.242565 3178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:45.243736 3160 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:45.243846 3160 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
format_stamp: "Formatted at 2025-06-25 08:04:25 on dist-test-slave-4qtp"
I20250625 08:04:45.244333 3160 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:45.273658 3160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:45.274356 3160 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:45.274565 3160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:45.281791 3160 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.126:36875
I20250625 08:04:45.281800 3230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.126:36875 every 8 connection(s)
I20250625 08:04:45.283022 3160 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
I20250625 08:04:45.286911 3231 sys_catalog.cc:263] Verifying existing consensus state
I20250625 08:04:45.288528 3231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap starting.
I20250625 08:04:45.292886 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 3160
I20250625 08:04:45.297947 3234 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:45.298171 3234 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:45.299065 3234 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
I20250625 08:04:45.300267 3231 log.cc:826] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Log is configured to *not* fsync() on all Append() calls
W20250625 08:04:45.300558 2996 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:45.340799 3231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=79 overwritten=0 applied=79 ignored=0} inserts{seen=51 ignored=0} mutations{seen=67 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:45.341336 3231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap complete.
I20250625 08:04:45.345645 3231 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:45.346091 3231 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0b114bd68344b0bd2ee322bd5eccdc, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.346570 3231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 8.79, Last appended by leader: 79, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:45.347308 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:45.347441 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:45.347761 3231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:45.347790 3205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 9 candidate_status { last_received { term: 8 index: 79 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
I20250625 08:04:45.348150 3205 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 8.
I20250625 08:04:45.348670 2999 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22
I20250625 08:04:45.349043 3234 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250625 08:04:45.349112 3234 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:45.349171 3234 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 8 FOLLOWER]: Advancing to term 9
I20250625 08:04:45.350382 3234 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:45.350857 3234 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 election: Requested vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
W20250625 08:04:45.352118 2996 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:45.351233 3205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 9 candidate_status { last_received { term: 8 index: 79 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
I20250625 08:04:45.352638 3205 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 8 FOLLOWER]: Advancing to term 9
I20250625 08:04:45.355960 3254 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:45.356036 3254 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: loading cluster ID for follower catalog manager: success
I20250625 08:04:45.358769 3205 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 9.
I20250625 08:04:45.359454 3254 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: acquiring CA information for follower catalog manager: success
I20250625 08:04:45.360560 2999 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22
I20250625 08:04:45.360771 3234 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 FOLLOWER]: Leader election won for term 9
I20250625 08:04:45.361018 3234 raft_consensus.cc:695] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 LEADER]: Becoming Leader. State: Replica: 755093bec4b54a9588598f814c3708e1, State: Running, Role: LEADER
I20250625 08:04:45.361346 3234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 8.79, Last appended by leader: 79, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:45.361644 3254 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:45.362671 3234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 9 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:45.362881 3234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:45.363236 3260 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:45.368372 3160 master_runner.cc:186] Error getting master registration for 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111),
I20250625 08:04:45.370674 3160 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:45.371539 3260 catalog_manager.cc:671] Loaded metadata for table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]
I20250625 08:04:45.372100 3260 catalog_manager.cc:671] Loaded metadata for table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]
I20250625 08:04:45.372466 3260 catalog_manager.cc:671] Loaded metadata for table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]
I20250625 08:04:45.372731 3260 catalog_manager.cc:671] Loaded metadata for table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]
I20250625 08:04:45.373063 3260 catalog_manager.cc:671] Loaded metadata for table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]
I20250625 08:04:45.373308 3260 catalog_manager.cc:671] Loaded metadata for table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]
I20250625 08:04:45.373545 3260 catalog_manager.cc:671] Loaded metadata for table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]
I20250625 08:04:45.373745 3260 catalog_manager.cc:671] Loaded metadata for table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]
I20250625 08:04:45.375104 3260 tablet_loader.cc:96] loaded metadata for tablet 0561d480e8b64f749c705741c3189f7f (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020])
I20250625 08:04:45.375881 3260 tablet_loader.cc:96] loaded metadata for tablet 2fb1337f4e44450bbf1379ecddae8bcf (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875])
I20250625 08:04:45.376243 3260 tablet_loader.cc:96] loaded metadata for tablet 42cc323e9e8a4b6f9f58b577c9057d2a (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a])
I20250625 08:04:45.376739 3260 tablet_loader.cc:96] loaded metadata for tablet 782a60aff92a4636bc98813f2214d1af (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206])
I20250625 08:04:45.376917 3260 tablet_loader.cc:96] loaded metadata for tablet 90154412b7b04b6db02f912a53000edb (table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904])
I20250625 08:04:45.377316 3260 tablet_loader.cc:96] loaded metadata for tablet a0db2bca14734321aa2167a9aade37e0 (table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d])
I20250625 08:04:45.377640 3260 tablet_loader.cc:96] loaded metadata for tablet b97c4e286a5040b1ad425242d9cb2dea (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6])
I20250625 08:04:45.378276 3260 tablet_loader.cc:96] loaded metadata for tablet ddb06adb0b7d4f888e489eb5721321fc (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089])
I20250625 08:04:45.378758 3260 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:45.379536 3260 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:45.379586 3260 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:45.380364 3260 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:45.380950 3260 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Loaded TSK: 0
I20250625 08:04:45.381692 3260 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:45.382269 3260 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:45.382874 3260 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 20
I20250625 08:04:45.426367 3011 master_service.cc:948] ReplaceTablet: received request to replace tablet ddb06adb0b7d4f888e489eb5721321fc from {username='slave'} at 127.0.0.1:39302
I20250625 08:04:45.426510 3012 master_service.cc:948] ReplaceTablet: received request to replace tablet 42cc323e9e8a4b6f9f58b577c9057d2a from {username='slave'} at 127.0.0.1:39302
I20250625 08:04:45.426731 3010 master_service.cc:948] ReplaceTablet: received request to replace tablet 2fb1337f4e44450bbf1379ecddae8bcf from {username='slave'} at 127.0.0.1:39302
I20250625 08:04:45.427981 3009 master_service.cc:948] ReplaceTablet: received request to replace tablet 2fb1337f4e44450bbf1379ecddae8bcf from {username='slave'} at 127.0.0.1:39302
I20250625 08:04:45.428146 3008 master_service.cc:948] ReplaceTablet: received request to replace tablet 2fb1337f4e44450bbf1379ecddae8bcf from {username='slave'} at 127.0.0.1:39302
I20250625 08:04:45.428401 3007 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39302:
name: "default.Table_d78071777f3646548f7f75cea674ecdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:45.428534 3003 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39302:
name: "default.Table_bd43623a5f9f40c58ad750234777ba9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:45.428653 3005 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39302:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:45.428759 3005 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39302:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:45.428956 3006 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39302:
name: "default.Table_28db8798a1444ed2b1d46a5a3eba537b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:45.429209 3004 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39302:
name: "default.Table_98189dd9ad454e5d8ee6edd848927c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 08:04:45.432842 2996 consensus_peers.cc:487] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 -> Peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Couldn't send request to peer 0404561252194ce29bf2924fd5e67e22. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:45.433403 3205 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Refusing update from remote peer 755093bec4b54a9588598f814c3708e1: Log matching property violated. Preceding OpId in replica: term: 8 index: 79. Preceding OpId from leader: term: 9 index: 82. (index mismatch)
I20250625 08:04:45.434001 3262 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
F20250625 08:04:45.436007 3065 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 4 out of 11 received events: NotificationEvent(eventId=21, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_31023e55a4384ac88f8415871e52d142, message={"server":"","servicePrincipal":"","db":"default","table":"table_31023e55a4384ac88f8415871e52d142","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_31023e55a4384ac88f8415871e52d142\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_31023e55a4384ac88f8415871e52d142\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_31023e55a4384ac88f8415871e52d142\",\"kudu.table_id\":\"7825ea67af2d42b68200097030b08116\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1750838669, eventType=ALTER_TABLE, dbName=default, tableName=table_7cb71508c2934e1aa091c71056ae7a01, message={"server":"","servicePrincipal":"","db":"default","table":"table_b49cbf7ee55042b39d7b5d3946a35618","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_b49cbf7ee55042b39d7b5d3946a35618\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_b49cbf7ee55042b39d7b5d3946a35618\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b49cbf7ee55042b39d7b5d3946a35618\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7cb71508c2934e1aa091c71056ae7a01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_7cb71508c2934e1aa091c71056ae7a01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7cb71508c2934e1aa091c71056ae7a01\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_d78071777f3646548f7f75cea674ecdc, message={"server":"","servicePrincipal":"","db":"default","table":"table_d78071777f3646548f7f75cea674ecdc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d78071777f3646548f7f75cea674ecdc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_d78071777f3646548f7f75cea674ecdc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d78071777f3646548f7f75cea674ecdc\",\"kudu.table_id\":\"0e43d85f40e545d9b31c5cb3e62f89a8\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=CREATE_TABLE, dbName=default, tableName=table_88079d64160a4318bf8ea1807cb3a30d, message={"server":"","servicePrincipal":"","db":"default","table":"table_88079d64160a4318bf8ea1807cb3a30d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_88079d64160a4318bf8ea1807cb3a30d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_88079d64160a4318bf8ea1807cb3a30d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_88079d64160a4318bf8ea1807cb3a30d\",\"kudu.table_id\":\"7c7e3cbdaf064697a864873ed87a21b6\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250625 08:04:45.438174 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 9 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:45.438323 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f91bf8d7dcd google::LogMessage::Fail() at ??:0
I20250625 08:04:45.440797 3234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:45.440922 3234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
@ 0x7f91bf8dbb93 google::LogMessage::SendToLog() at ??:0
I20250625 08:04:45.441967 3264 mvcc.cc:204] Tried to move back new op lower bound from 7171435255519969280 to 7171435255243440128. Current Snapshot: MvccSnapshot[applied={T|T < 7171435255519969280}]
I20250625 08:04:45.442430 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:45.442555 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:45.443006 3011 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet ddb06adb0b7d4f888e489eb5721321fc
I20250625 08:04:45.443486 3010 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet 2fb1337f4e44450bbf1379ecddae8bcf
I20250625 08:04:45.443987 3012 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet 42cc323e9e8a4b6f9f58b577c9057d2a
I20250625 08:04:45.444245 3012 catalog_manager.cc:6652] ReplaceTablet: tablet 42cc323e9e8a4b6f9f58b577c9057d2a deleted and replaced by tablet 499fa735a33d450db3047beb0e018f6f
I20250625 08:04:45.444499 3012 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39302:
name: "default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:45.444514 3009 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39302:
table { table_name: "default.Table_b49cbf7ee55042b39d7b5d3946a35618" } new_table_name: "default.Table_7cb71508c2934e1aa091c71056ae7a01" modify_external_catalogs: true
I20250625 08:04:45.444012 3010 catalog_manager.cc:6652] ReplaceTablet: tablet 2fb1337f4e44450bbf1379ecddae8bcf deleted and replaced by tablet 39645fa894654da6949adbe9e3d1dc96
I20250625 08:04:45.445313 3010 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39302:
table { table_name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2" } modify_external_catalogs: true
I20250625 08:04:45.445432 3010 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39302:
table { table_name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2" } modify_external_catalogs: true
I20250625 08:04:45.445566 3011 catalog_manager.cc:6652] ReplaceTablet: tablet ddb06adb0b7d4f888e489eb5721321fc deleted and replaced by tablet cb5f968f985c422bbf615c31fd936ef1
I20250625 08:04:45.445726 3011 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39302:
name: "default.Table_128075ffc1db4314b52b7c28536f6e89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f91bf8d77cc google::LogMessage::Flush() at ??:0
@ 0x7f91bf8d7a11 google::LogMessage::~LogMessage() at ??:0
I20250625 08:04:45.450482 3008 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39302:
name: "default.Table_174089cfe43b4dbb9ed1b9c2921f9187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:45.455631 1827 tablet_service.cc:1515] Processing DeleteTablet for tablet 2fb1337f4e44450bbf1379ecddae8bcf with delete_type TABLET_DATA_DELETED (replaced by tablet 39645fa894654da6949adbe9e3d1dc96) from {username='slave'} at 127.0.0.1:53300
I20250625 08:04:45.455801 1823 tablet_service.cc:1515] Processing DeleteTablet for tablet 42cc323e9e8a4b6f9f58b577c9057d2a with delete_type TABLET_DATA_DELETED (replaced by tablet 499fa735a33d450db3047beb0e018f6f) from {username='slave'} at 127.0.0.1:53300
I20250625 08:04:45.455933 1825 tablet_service.cc:1515] Processing DeleteTablet for tablet ddb06adb0b7d4f888e489eb5721321fc with delete_type TABLET_DATA_DELETED (replaced by tablet cb5f968f985c422bbf615c31fd936ef1) from {username='slave'} at 127.0.0.1:53300
I20250625 08:04:45.460237 1823 tablet_service.cc:1468] Processing CreateTablet for tablet 39645fa894654da6949adbe9e3d1dc96 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.460546 1823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39645fa894654da6949adbe9e3d1dc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.460507 1827 tablet_service.cc:1468] Processing CreateTablet for tablet 499fa735a33d450db3047beb0e018f6f (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.460705 1827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499fa735a33d450db3047beb0e018f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.460237 1825 tablet_service.cc:1468] Processing CreateTablet for tablet cb5f968f985c422bbf615c31fd936ef1 (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.460898 1825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5f968f985c422bbf615c31fd936ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.463605 3278 tablet_replica.cc:331] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:45.463737 3278 raft_consensus.cc:2241] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:45.463798 1958 tablet_service.cc:1515] Processing DeleteTablet for tablet 2fb1337f4e44450bbf1379ecddae8bcf with delete_type TABLET_DATA_DELETED (replaced by tablet 39645fa894654da6949adbe9e3d1dc96) from {username='slave'} at 127.0.0.1:56374
I20250625 08:04:45.463860 3278 raft_consensus.cc:2270] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:45.464017 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet 42cc323e9e8a4b6f9f58b577c9057d2a with delete_type TABLET_DATA_DELETED (replaced by tablet 499fa735a33d450db3047beb0e018f6f) from {username='slave'} at 127.0.0.1:56374
I20250625 08:04:45.464267 3278 ts_tablet_manager.cc:1905] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:45.464226 1961 tablet_service.cc:1468] Processing CreateTablet for tablet 39645fa894654da6949adbe9e3d1dc96 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.464309 1960 tablet_service.cc:1468] Processing CreateTablet for tablet cb5f968f985c422bbf615c31fd936ef1 (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.464481 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39645fa894654da6949adbe9e3d1dc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.464655 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5f968f985c422bbf615c31fd936ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.464813 1959 tablet_service.cc:1468] Processing CreateTablet for tablet 499fa735a33d450db3047beb0e018f6f (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.465015 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499fa735a33d450db3047beb0e018f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.466323 3280 tablet_bootstrap.cc:492] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:45.464039 1963 tablet_service.cc:1515] Processing DeleteTablet for tablet ddb06adb0b7d4f888e489eb5721321fc with delete_type TABLET_DATA_DELETED (replaced by tablet cb5f968f985c422bbf615c31fd936ef1) from {username='slave'} at 127.0.0.1:56374
I20250625 08:04:45.467384 3280 tablet_bootstrap.cc:654] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.468048 3278 ts_tablet_manager.cc:1918] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:45.468111 3278 log.cc:1199] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/2fb1337f4e44450bbf1379ecddae8bcf
I20250625 08:04:45.468389 3278 ts_tablet_manager.cc:1939] T 2fb1337f4e44450bbf1379ecddae8bcf P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:45.469089 3278 tablet_replica.cc:331] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:45.469074 2999 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 2fb1337f4e44450bbf1379ecddae8bcf (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:45.469200 3278 raft_consensus.cc:2241] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:45.469379 3278 raft_consensus.cc:2270] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:45.469945 1961 tablet_service.cc:1468] Processing CreateTablet for tablet a0db2bca14734321aa2167a9aade37e0 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.469945 1823 tablet_service.cc:1468] Processing CreateTablet for tablet a0db2bca14734321aa2167a9aade37e0 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.470160 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0db2bca14734321aa2167a9aade37e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.470160 1823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0db2bca14734321aa2167a9aade37e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.472469 3280 tablet_bootstrap.cc:492] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:45.472569 3280 ts_tablet_manager.cc:1397] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 08:04:45.473055 3280 raft_consensus.cc:357] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.473158 3280 raft_consensus.cc:383] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.473186 3280 raft_consensus.cc:738] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.473307 3280 consensus_queue.cc:260] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.473572 3280 ts_tablet_manager.cc:1428] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:45.473718 3280 tablet_bootstrap.cc:492] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:45.474628 3278 ts_tablet_manager.cc:1905] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f91c4870c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 08:04:45.477062 3278 ts_tablet_manager.cc:1918] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:45.477128 3278 log.cc:1199] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/42cc323e9e8a4b6f9f58b577c9057d2a
I20250625 08:04:45.477380 3278 ts_tablet_manager.cc:1939] T 42cc323e9e8a4b6f9f58b577c9057d2a P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:45.477968 3278 tablet_replica.cc:331] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:45.478091 3278 raft_consensus.cc:2241] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:45.478255 3278 raft_consensus.cc:2270] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:45.478639 3278 ts_tablet_manager.cc:1905] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:45.478745 2999 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 42cc323e9e8a4b6f9f58b577c9057d2a (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]) successfully deleted
I20250625 08:04:45.479542 1895 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:45.479763 3282 tablet_bootstrap.cc:492] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:45.480350 1687 tablet_service.cc:1515] Processing DeleteTablet for tablet ddb06adb0b7d4f888e489eb5721321fc with delete_type TABLET_DATA_DELETED (replaced by tablet cb5f968f985c422bbf615c31fd936ef1) from {username='slave'} at 127.0.0.1:60742
I20250625 08:04:45.480635 3278 ts_tablet_manager.cc:1918] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:45.480705 3278 log.cc:1199] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/ddb06adb0b7d4f888e489eb5721321fc
I20250625 08:04:45.480751 1685 tablet_service.cc:1515] Processing DeleteTablet for tablet 2fb1337f4e44450bbf1379ecddae8bcf with delete_type TABLET_DATA_DELETED (replaced by tablet 39645fa894654da6949adbe9e3d1dc96) from {username='slave'} at 127.0.0.1:60742
I20250625 08:04:45.480808 1687 tablet_service.cc:1468] Processing CreateTablet for tablet 499fa735a33d450db3047beb0e018f6f (DEFAULT_TABLE table=default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.480911 1686 tablet_service.cc:1515] Processing DeleteTablet for tablet 42cc323e9e8a4b6f9f58b577c9057d2a with delete_type TABLET_DATA_DELETED (replaced by tablet 499fa735a33d450db3047beb0e018f6f) from {username='slave'} at 127.0.0.1:60742
I20250625 08:04:45.480978 3278 ts_tablet_manager.cc:1939] T ddb06adb0b7d4f888e489eb5721321fc P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:45.481074 1687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499fa735a33d450db3047beb0e018f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.481175 1688 tablet_service.cc:1468] Processing CreateTablet for tablet cb5f968f985c422bbf615c31fd936ef1 (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.481427 1688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb5f968f985c422bbf615c31fd936ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.481415 3195 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" instance_seqno: 1750838667017582) as {username='slave'} at 127.1.19.66:48053; Asking this server to re-register.
I20250625 08:04:45.481988 1895 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:45.482688 1689 tablet_service.cc:1468] Processing CreateTablet for tablet 39645fa894654da6949adbe9e3d1dc96 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.482936 1689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39645fa894654da6949adbe9e3d1dc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.483223 3195 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:45.483489 1691 tablet_service.cc:1468] Processing CreateTablet for tablet a0db2bca14734321aa2167a9aade37e0 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:45.483688 1691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0db2bca14734321aa2167a9aade37e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:45.484865 3282 tablet_bootstrap.cc:654] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.485352 3280 tablet_bootstrap.cc:654] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.486095 2999 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet ddb06adb0b7d4f888e489eb5721321fc (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:45.486383 3285 tablet_bootstrap.cc:492] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:45.487330 3285 tablet_bootstrap.cc:654] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.487385 3280 tablet_bootstrap.cc:492] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:45.487473 3280 ts_tablet_manager.cc:1397] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250625 08:04:45.488078 3280 raft_consensus.cc:357] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.488204 3280 raft_consensus.cc:383] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.488243 3280 raft_consensus.cc:738] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.488404 3280 consensus_queue.cc:260] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.488719 3280 ts_tablet_manager.cc:1428] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:45.488847 3281 tablet_replica.cc:331] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:45.488968 3281 raft_consensus.cc:2241] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:45.489137 3281 raft_consensus.cc:2270] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:45.489583 3281 ts_tablet_manager.cc:1905] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:45.490662 3280 tablet_bootstrap.cc:492] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:45.492048 3280 tablet_bootstrap.cc:654] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
@ 0x7f91c486fba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 08:04:45.493299 3280 tablet_bootstrap.cc:492] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:45.493381 3280 ts_tablet_manager.cc:1397] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:45.493955 3280 raft_consensus.cc:357] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.494076 3280 raft_consensus.cc:383] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.494117 3280 raft_consensus.cc:738] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.494263 3280 consensus_queue.cc:260] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.494621 3280 ts_tablet_manager.cc:1428] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 08:04:45.494830 3280 tablet_bootstrap.cc:492] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:45.495076 3282 tablet_bootstrap.cc:492] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:45.495167 3282 ts_tablet_manager.cc:1397] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250625 08:04:45.495806 3280 tablet_bootstrap.cc:654] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.495740 3282 raft_consensus.cc:357] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.495849 3282 raft_consensus.cc:383] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.495880 3282 raft_consensus.cc:738] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.496006 3282 consensus_queue.cc:260] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.496359 3281 ts_tablet_manager.cc:1918] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:45.496431 3281 log.cc:1199] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/2fb1337f4e44450bbf1379ecddae8bcf
I20250625 08:04:45.496729 3281 ts_tablet_manager.cc:1939] T 2fb1337f4e44450bbf1379ecddae8bcf P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:45.497712 2999 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet 2fb1337f4e44450bbf1379ecddae8bcf (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:45.497790 3282 ts_tablet_manager.cc:1428] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 08:04:45.497952 3281 tablet_replica.cc:331] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:45.498071 3281 raft_consensus.cc:2241] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:45.498160 3281 raft_consensus.cc:2270] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:45.497970 3282 tablet_bootstrap.cc:492] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:45.498536 3281 ts_tablet_manager.cc:1905] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:45.498675 3284 tablet_replica.cc:331] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:45.498817 3284 raft_consensus.cc:2241] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:45.498946 3284 raft_consensus.cc:2270] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:45.499353 3282 tablet_bootstrap.cc:654] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.499334 3284 ts_tablet_manager.cc:1905] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:45.500622 3281 ts_tablet_manager.cc:1918] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:45.500697 3281 log.cc:1199] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/42cc323e9e8a4b6f9f58b577c9057d2a
I20250625 08:04:45.500954 3281 ts_tablet_manager.cc:1939] T 42cc323e9e8a4b6f9f58b577c9057d2a P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:45.501436 2030 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:45.501703 3284 ts_tablet_manager.cc:1918] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:45.501670 2999 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet 42cc323e9e8a4b6f9f58b577c9057d2a (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]) successfully deleted
I20250625 08:04:45.501770 3284 log.cc:1199] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/ddb06adb0b7d4f888e489eb5721321fc
I20250625 08:04:45.502074 3284 ts_tablet_manager.cc:1939] T ddb06adb0b7d4f888e489eb5721321fc P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:45.502799 3284 tablet_replica.cc:331] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:45.502806 2999 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet ddb06adb0b7d4f888e489eb5721321fc (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:45.502943 3284 raft_consensus.cc:2241] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:45.503125 3284 raft_consensus.cc:2270] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:45.503477 3284 ts_tablet_manager.cc:1905] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f91c486f30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 08:04:45.504009 3195 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" instance_seqno: 1750838667205184) as {username='slave'} at 127.1.19.67:59699; Asking this server to re-register.
I20250625 08:04:45.504549 2030 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:45.505573 3284 ts_tablet_manager.cc:1918] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:45.505605 3195 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:45.505637 3284 log.cc:1199] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/2fb1337f4e44450bbf1379ecddae8bcf
I20250625 08:04:45.505872 3284 ts_tablet_manager.cc:1939] T 2fb1337f4e44450bbf1379ecddae8bcf P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:45.506529 3284 tablet_replica.cc:331] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:45.506646 3284 raft_consensus.cc:2241] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:45.506747 3284 raft_consensus.cc:2270] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:45.507112 3284 ts_tablet_manager.cc:1905] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:45.509009 3284 ts_tablet_manager.cc:1918] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:45.509083 3284 log.cc:1199] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/42cc323e9e8a4b6f9f58b577c9057d2a
I20250625 08:04:45.509328 3284 ts_tablet_manager.cc:1939] T 42cc323e9e8a4b6f9f58b577c9057d2a P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:45.509768 3285 tablet_bootstrap.cc:492] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:45.509845 3285 ts_tablet_manager.cc:1397] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20250625 08:04:45.510293 3285 raft_consensus.cc:357] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.510386 3285 raft_consensus.cc:383] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.510412 3285 raft_consensus.cc:738] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.510566 3285 consensus_queue.cc:260] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.510730 2999 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 2fb1337f4e44450bbf1379ecddae8bcf (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:45.510850 3285 ts_tablet_manager.cc:1428] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:45.510923 2999 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 42cc323e9e8a4b6f9f58b577c9057d2a (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]) successfully deleted
I20250625 08:04:45.510941 3281 tablet_replica.cc:331] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:45.511058 3285 tablet_bootstrap.cc:492] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:45.511065 3281 raft_consensus.cc:2241] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:45.511179 3281 raft_consensus.cc:2270] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:45.511552 3281 ts_tablet_manager.cc:1905] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:45.512068 3285 tablet_bootstrap.cc:654] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.512416 3282 tablet_bootstrap.cc:492] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:45.512492 3282 ts_tablet_manager.cc:1397] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250625 08:04:45.512992 3282 raft_consensus.cc:357] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.513088 3282 raft_consensus.cc:383] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.513118 3282 raft_consensus.cc:738] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.513288 3282 consensus_queue.cc:260] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.513566 3282 ts_tablet_manager.cc:1428] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:45.513720 3281 ts_tablet_manager.cc:1918] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:45.513774 3281 log.cc:1199] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/ddb06adb0b7d4f888e489eb5721321fc
I20250625 08:04:45.514042 3281 ts_tablet_manager.cc:1939] T ddb06adb0b7d4f888e489eb5721321fc P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:45.515084 2999 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet ddb06adb0b7d4f888e489eb5721321fc (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:45.513779 3282 tablet_bootstrap.cc:492] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
@ 0x7f91c4874a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250625 08:04:45.516502 3282 tablet_bootstrap.cc:654] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.517082 3280 tablet_bootstrap.cc:492] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:45.517172 3280 ts_tablet_manager.cc:1397] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
@ 0x55c6ffc28752 std::function<>::operator()() at ??:0
I20250625 08:04:45.517670 3280 raft_consensus.cc:357] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.517777 3280 raft_consensus.cc:383] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.517808 3280 raft_consensus.cc:738] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.517935 3280 consensus_queue.cc:260] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.518219 3280 ts_tablet_manager.cc:1428] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:45.520401 1758 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:45.520529 1758 heartbeater.cc:499] Master 127.1.19.126:36875 was elected leader, sending a full tablet report...
@ 0x7f91c0d290b7 kudu::Thread::SuperviseThread() at ??:0
I20250625 08:04:45.521045 3282 tablet_bootstrap.cc:492] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:45.521153 3282 ts_tablet_manager.cc:1397] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250625 08:04:45.521250 3285 tablet_bootstrap.cc:492] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:45.521345 3285 ts_tablet_manager.cc:1397] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250625 08:04:45.521824 3195 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" instance_seqno: 1750838666815429) as {username='slave'} at 127.1.19.65:47381; Asking this server to re-register.
I20250625 08:04:45.521817 3282 raft_consensus.cc:357] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.521850 3285 raft_consensus.cc:357] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.521939 3282 raft_consensus.cc:383] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.521957 3285 raft_consensus.cc:383] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.521991 3285 raft_consensus.cc:738] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.521981 3282 raft_consensus.cc:738] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.522119 3285 consensus_queue.cc:260] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.522135 3282 consensus_queue.cc:260] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.522419 3285 ts_tablet_manager.cc:1428] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:45.522507 3282 ts_tablet_manager.cc:1428] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 08:04:45.522661 3285 tablet_bootstrap.cc:492] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:45.522751 3282 tablet_bootstrap.cc:492] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
@ 0x7f91c15186db start_thread at ??:0
I20250625 08:04:45.523770 1758 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:45.523986 3285 tablet_bootstrap.cc:654] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.524051 3282 tablet_bootstrap.cc:654] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.524952 3195 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
@ 0x7f91bea8e71f clone at ??:0
I20250625 08:04:45.525434 3285 tablet_bootstrap.cc:492] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:45.525516 3285 ts_tablet_manager.cc:1397] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250625 08:04:45.526139 3285 raft_consensus.cc:357] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.526257 3285 raft_consensus.cc:383] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.526294 3285 raft_consensus.cc:738] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.526480 3285 consensus_queue.cc:260] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.526794 3285 ts_tablet_manager.cc:1428] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:45.527009 3285 tablet_bootstrap.cc:492] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:45.528090 3285 tablet_bootstrap.cc:654] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:45.530536 3285 tablet_bootstrap.cc:492] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:45.530627 3285 ts_tablet_manager.cc:1397] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 08:04:45.530728 3282 tablet_bootstrap.cc:492] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:45.530814 3282 ts_tablet_manager.cc:1397] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250625 08:04:45.531126 3285 raft_consensus.cc:357] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.531224 3285 raft_consensus.cc:383] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.531256 3285 raft_consensus.cc:738] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.531275 3282 raft_consensus.cc:357] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.531376 3282 raft_consensus.cc:383] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:45.531407 3282 raft_consensus.cc:738] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:45.531426 3285 consensus_queue.cc:260] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.531520 3282 consensus_queue.cc:260] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.531816 3282 ts_tablet_manager.cc:1428] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 08:04:45.533937 3285 ts_tablet_manager.cc:1428] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250625 08:04:45.641094 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 11: this message will repeat every 5th retry.
I20250625 08:04:45.654523 3141 raft_consensus.cc:491] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:45.654659 3141 raft_consensus.cc:513] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.655212 3141 leader_election.cc:290] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:45.655570 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb5f968f985c422bbf615c31fd936ef1" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:45.655715 1847 raft_consensus.cc:2466] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:45.655776 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb5f968f985c422bbf615c31fd936ef1" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:45.655941 1711 raft_consensus.cc:2466] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:45.656035 1919 leader_election.cc:304] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:45.656219 3141 raft_consensus.cc:2802] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:45.656278 3141 raft_consensus.cc:491] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:45.656319 3141 raft_consensus.cc:3058] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.657191 3141 raft_consensus.cc:513] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.657523 3141 leader_election.cc:290] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:45.657876 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb5f968f985c422bbf615c31fd936ef1" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:45.657922 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb5f968f985c422bbf615c31fd936ef1" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:45.658037 1847 raft_consensus.cc:3058] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.658054 1711 raft_consensus.cc:3058] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.658883 1847 raft_consensus.cc:2466] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:45.659163 1711 raft_consensus.cc:2466] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:45.659178 1919 leader_election.cc:304] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:45.659464 3141 raft_consensus.cc:2802] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:45.659677 3141 raft_consensus.cc:695] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:45.659804 3141 consensus_queue.cc:237] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.667384 2930 raft_consensus.cc:491] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:45.667557 2930 raft_consensus.cc:513] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.668169 2930 leader_election.cc:290] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:45.668642 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499fa735a33d450db3047beb0e018f6f" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:45.668689 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499fa735a33d450db3047beb0e018f6f" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:45.668779 1982 raft_consensus.cc:2466] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:45.668841 1711 raft_consensus.cc:2466] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:45.669117 1784 leader_election.cc:304] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:45.669275 2930 raft_consensus.cc:2802] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:45.669337 2930 raft_consensus.cc:491] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:45.669378 2930 raft_consensus.cc:3058] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.670233 2930 raft_consensus.cc:513] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.670614 2930 leader_election.cc:290] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:45.670951 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499fa735a33d450db3047beb0e018f6f" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:45.671016 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "499fa735a33d450db3047beb0e018f6f" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:45.671067 1982 raft_consensus.cc:3058] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.671140 1711 raft_consensus.cc:3058] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.671821 1982 raft_consensus.cc:2466] T 499fa735a33d450db3047beb0e018f6f P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:45.672174 1711 raft_consensus.cc:2466] T 499fa735a33d450db3047beb0e018f6f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:45.672183 1784 leader_election.cc:304] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:45.672392 2930 raft_consensus.cc:2802] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:45.672601 2930 raft_consensus.cc:695] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:45.672752 2930 consensus_queue.cc:237] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
W20250625 08:04:45.702827 2037 connection.cc:537] client connection to 127.1.19.124:37745 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:45.703359 2037 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.19.124:37745)
W20250625 08:04:45.703775 2037 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.124:37745)
W20250625 08:04:45.703893 2037 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.19.124:37745)
W20250625 08:04:45.704007 2037 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.19.124:37745)
W20250625 08:04:45.705200 3161 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:45.703425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.124:37745 (local address 127.1.19.67:51577)
0625 08:04:45.703641 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:45.703658 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:45.703758 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:45.705101 (+ 1343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":117}
W20250625 08:04:45.705688 2929 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:45.705000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.124:37745 (local address 127.0.0.1:39312)
0625 08:04:45.705104 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:45.705115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:45.705183 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:45.705581 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":45}
W20250625 08:04:45.705679 3164 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:45.703631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.124:37745 (local address 127.1.19.66:45357)
0625 08:04:45.703807 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:45.703823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:45.703905 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:45.705577 (+ 1672us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":97}
W20250625 08:04:45.705726 3139 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:45.704395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.124:37745 (local address 127.1.19.65:33079)
0625 08:04:45.704632 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:45.704647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:45.704729 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:45.705588 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":152}
I20250625 08:04:45.760970 2930 raft_consensus.cc:491] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:45.761126 2930 raft_consensus.cc:513] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.761510 2930 leader_election.cc:290] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:45.761929 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0db2bca14734321aa2167a9aade37e0" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:45.761945 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0db2bca14734321aa2167a9aade37e0" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:45.762061 1982 raft_consensus.cc:2466] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:45.762073 1711 raft_consensus.cc:2466] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:45.762356 1784 leader_election.cc:304] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:45.762562 2930 raft_consensus.cc:2802] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:45.762621 2930 raft_consensus.cc:491] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:45.762676 2930 raft_consensus.cc:3058] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.763515 2930 raft_consensus.cc:513] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.763851 2930 leader_election.cc:290] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:45.764168 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0db2bca14734321aa2167a9aade37e0" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:45.764221 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0db2bca14734321aa2167a9aade37e0" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:45.764282 1982 raft_consensus.cc:3058] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.764354 1711 raft_consensus.cc:3058] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.765064 1982 raft_consensus.cc:2466] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:45.765075 1711 raft_consensus.cc:2466] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:45.765369 1784 leader_election.cc:304] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:45.765529 2930 raft_consensus.cc:2802] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:45.765625 2930 raft_consensus.cc:695] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:45.765762 2930 consensus_queue.cc:237] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.775101 2661 raft_consensus.cc:491] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:45.775245 2661 raft_consensus.cc:513] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.775777 2661 leader_election.cc:290] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:45.776266 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39645fa894654da6949adbe9e3d1dc96" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:45.776297 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39645fa894654da6949adbe9e3d1dc96" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:45.776409 1982 raft_consensus.cc:2466] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:45.776419 1847 raft_consensus.cc:2466] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:45.776714 1647 leader_election.cc:304] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:45.776901 2661 raft_consensus.cc:2802] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:45.776958 2661 raft_consensus.cc:491] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:45.777014 2661 raft_consensus.cc:3058] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.777931 2661 raft_consensus.cc:513] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.778262 2661 leader_election.cc:290] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:45.778785 1982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39645fa894654da6949adbe9e3d1dc96" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:45.778802 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39645fa894654da6949adbe9e3d1dc96" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:45.778928 1847 raft_consensus.cc:3058] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.778928 1982 raft_consensus.cc:3058] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:45.779922 1847 raft_consensus.cc:2466] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:45.779922 1982 raft_consensus.cc:2466] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:45.780229 1647 leader_election.cc:304] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:45.780392 2661 raft_consensus.cc:2802] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:45.780483 2661 raft_consensus.cc:695] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:45.780654 2661 consensus_queue.cc:237] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:45.957041 3313 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:45.957182 3313 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:45.958361 3313 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:45.958947 3179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:45.959553 3180 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:45.959646 3179 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:45.959739 3179 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:45.959928 3313 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250625 08:04:46.097131 3308 consensus_queue.cc:1035] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:46.100852 2930 consensus_queue.cc:1035] T 499fa735a33d450db3047beb0e018f6f P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:46.174587 3141 consensus_queue.cc:1035] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:46.178259 3325 consensus_queue.cc:1035] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:46.215802 2731 consensus_queue.cc:1035] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:46.220561 2731 consensus_queue.cc:1035] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:46.222671 2930 consensus_queue.cc:1035] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 08:04:46.225453 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 11: this message will repeat every 5th retry.
I20250625 08:04:46.228652 2930 consensus_queue.cc:1035] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:46.535445 3332 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:46.535588 3332 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:46.536075 3332 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:46.537456 3180 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:46.537534 3179 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:46.537595 3179 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:46.537885 3332 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250625 08:04:47.479447 3343 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:47.479583 3343 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:47.480050 3343 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:47.481397 3180 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:47.481498 3179 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:47.481585 3179 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:47.481719 3343 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250625 08:04:47.654479 1101 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu with pid 3160
I20250625 08:04:47.683573 2731 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:47.683701 2731 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:47.684126 2731 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:04:47.684741 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:04:47.684860 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:47.684899 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:47.685040 2731 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:48.207759 2732 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:48.207896 2732 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:48.208338 2732 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:04:48.209074 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:04:48.209200 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:04:48.209239 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:48.209383 2732 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:48.457391 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.126:36875
--webserver_interface=127.1.19.126
--webserver_port=32959
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 08:04:48.565553 3363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:48.565786 3363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:48.565825 3363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:48.569000 3363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:48.569062 3363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:48.569083 3363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:48.569101 3363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:48.572832 3363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.126:36875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.19.126
--webserver_port=32959
--never_fsync=true
--heap_profile_path=/tmp/kudu.3363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:48.573942 3363 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:48.575110 3363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:48.581758 3368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:48.581804 3371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:48.581967 3363 server_base.cc:1048] running on GCE node
W20250625 08:04:48.582113 3369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:48.582617 3363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:48.583566 3363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:48.584710 3363 hybrid_clock.cc:648] HybridClock initialized: now 1750838688584687 us; error 39 us; skew 500 ppm
I20250625 08:04:48.586633 3363 webserver.cc:469] Webserver started at http://127.1.19.126:32959/ using document root <none> and password file <none>
I20250625 08:04:48.587188 3363 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:48.587249 3363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:48.591058 3363 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:48.593217 3377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:48.594230 3363 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:48.594337 3363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
format_stamp: "Formatted at 2025-06-25 08:04:25 on dist-test-slave-4qtp"
I20250625 08:04:48.594776 3363 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:48.608335 3363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:48.609032 3363 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:48.609212 3363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:48.616500 3363 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.126:36875
I20250625 08:04:48.616505 3432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.126:36875 every 8 connection(s)
I20250625 08:04:48.617646 3363 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-0/data/info.pb
I20250625 08:04:48.621560 3433 sys_catalog.cc:263] Verifying existing consensus state
I20250625 08:04:48.623291 3433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap starting.
I20250625 08:04:48.623659 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 3363
I20250625 08:04:48.635030 3433 log.cc:826] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:48.676564 3433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=84 ignored=0} inserts{seen=54 ignored=0} mutations{seen=74 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:48.677032 3433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Bootstrap complete.
I20250625 08:04:48.679790 3433 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:48.680150 3433 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0b114bd68344b0bd2ee322bd5eccdc, State: Initialized, Role: FOLLOWER
I20250625 08:04:48.680619 3433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 9.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:48.681197 3437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:48.681317 3437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:48.681782 3433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:48.688191 3451 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:48.688278 3451 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: loading cluster ID for follower catalog manager: success
I20250625 08:04:48.691042 3451 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: acquiring CA information for follower catalog manager: success
I20250625 08:04:48.692824 3451 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:48.701246 1758 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:48.701122 3363 master_runner.cc:186] Error getting master registration for 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111),
I20250625 08:04:48.703213 3363 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:48.703918 3397 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" instance_seqno: 1750838666815429) as {username='slave'} at 127.1.19.65:45273; Asking this server to re-register.
I20250625 08:04:48.704561 1758 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:48.705977 3397 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:48.774034 1895 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:48.775162 3397 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" instance_seqno: 1750838667017582) as {username='slave'} at 127.1.19.66:47573; Asking this server to re-register.
I20250625 08:04:48.775609 1895 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:48.776423 3397 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:49.108099 2030 heartbeater.cc:344] Connected to a master server at 127.1.19.126:36875
I20250625 08:04:49.109346 3397 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" instance_seqno: 1750838667205184) as {username='slave'} at 127.1.19.67:33897; Asking this server to re-register.
I20250625 08:04:49.109737 2030 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:49.110605 3397 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:49.243711 3463 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:49.243925 3463 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:49.245168 3463 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:49.245750 3378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:49.246498 3378 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:49.246663 3379 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:49.246727 3379 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:49.246948 3463 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250625 08:04:49.403831 2731 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:49.403956 2731 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:04:49.404369 2731 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:04:49.405068 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:04:49.405176 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:04:49.405225 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:49.405390 2731 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:49.910851 3472 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:49.911039 3472 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:49.911720 3472 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:49.912992 3378 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:49.913084 3379 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:49.913144 3379 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:49.913283 3472 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250625 08:04:50.629284 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 16: this message will repeat every 5th retry.
I20250625 08:04:51.012545 3488 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:51.012689 3488 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:51.013171 3488 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:51.014427 3379 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:51.014806 3378 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:51.014886 3378 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:51.015038 3488 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250625 08:04:51.173296 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 16: this message will repeat every 5th retry.
I20250625 08:04:51.691615 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.124:37745
--webserver_interface=127.1.19.124
--webserver_port=44175
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 08:04:51.801275 3501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:51.801509 3501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:51.801563 3501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:51.804812 3501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:51.804881 3501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:51.804903 3501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:51.804926 3501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:51.809047 3501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.124:37745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.19.124
--webserver_port=44175
--never_fsync=true
--heap_profile_path=/tmp/kudu.3501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:51.810254 3501 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:51.811386 3501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:51.818128 3511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:51.818131 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:51.818109 3509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:51.818461 3501 server_base.cc:1048] running on GCE node
I20250625 08:04:51.818969 3501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:51.819860 3501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:51.821013 3501 hybrid_clock.cc:648] HybridClock initialized: now 1750838691820999 us; error 26 us; skew 500 ppm
I20250625 08:04:51.826478 3501 webserver.cc:469] Webserver started at http://127.1.19.124:44175/ using document root <none> and password file <none>
I20250625 08:04:51.827265 3501 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:51.827345 3501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:51.831637 3501 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20250625 08:04:51.833957 3517 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:51.835033 3501 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250625 08:04:51.835141 3501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
uuid: "755093bec4b54a9588598f814c3708e1"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:51.835554 3501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:51.846490 3501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:51.847177 3501 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:51.847357 3501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:51.854650 3501 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.124:37745
I20250625 08:04:51.854662 3570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.124:37745 every 8 connection(s)
I20250625 08:04:51.855791 3501 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-2/data/info.pb
I20250625 08:04:51.858153 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 3501
I20250625 08:04:51.859891 3571 sys_catalog.cc:263] Verifying existing consensus state
I20250625 08:04:51.861559 3571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap starting.
I20250625 08:04:51.870882 3571 log.cc:826] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:51.911581 3571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=84 ignored=0} inserts{seen=54 ignored=0} mutations{seen=74 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:51.912039 3571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Bootstrap complete.
I20250625 08:04:51.914758 3571 raft_consensus.cc:357] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:51.915131 3571 raft_consensus.cc:738] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755093bec4b54a9588598f814c3708e1, State: Initialized, Role: FOLLOWER
I20250625 08:04:51.915598 3571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 9.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:51.916091 3575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:51.916211 3575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:51.916704 3571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:51.924415 3589 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:51.924569 3589 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: loading cluster ID for follower catalog manager: success
I20250625 08:04:51.931437 3589 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: acquiring CA information for follower catalog manager: success
I20250625 08:04:51.933779 3589 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:51.934579 3501 master_runner.cc:186] Error getting master registration for 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111),
I20250625 08:04:51.936473 3501 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:52.062394 3494 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:04:52.062578 3494 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:04:52.063081 3494 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:04:52.063901 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:04:52.064080 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:04:52.064149 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:04:52.064363 3494 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:04:52.114735 2029 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:52.116650 3534 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" instance_seqno: 1750838667205184) as {username='slave'} at 127.1.19.67:50565; Asking this server to re-register.
I20250625 08:04:52.117182 2029 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:52.118338 3534 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:52.212795 1757 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:52.214135 3534 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" instance_seqno: 1750838666815429) as {username='slave'} at 127.1.19.65:48283; Asking this server to re-register.
I20250625 08:04:52.214777 1757 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:52.215958 3534 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:52.280751 1894 heartbeater.cc:344] Connected to a master server at 127.1.19.124:37745
I20250625 08:04:52.281862 3534 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" instance_seqno: 1750838667017582) as {username='slave'} at 127.1.19.66:51975; Asking this server to re-register.
I20250625 08:04:52.282338 1894 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:52.283231 3534 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:52.515498 3599 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:52.515735 3599 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:52.516785 3600 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:52.516896 3600 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:52.517030 3599 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
I20250625 08:04:52.517526 3407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 10 candidate_status { last_received { term: 9 index: 84 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
I20250625 08:04:52.517729 3600 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:52.517773 3407 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 9.
W20250625 08:04:52.517843 3518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:52.518258 3521 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters:
I20250625 08:04:52.518571 3599 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250625 08:04:52.518649 3599 raft_consensus.cc:491] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:52.518694 3599 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 9 FOLLOWER]: Advancing to term 10
W20250625 08:04:52.518846 3518 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
W20250625 08:04:52.518884 3378 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:52.520838 3599 raft_consensus.cc:513] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:52.521266 3599 leader_election.cc:290] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 10 election: Requested vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317)
I20250625 08:04:52.521526 3407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "755093bec4b54a9588598f814c3708e1" candidate_term: 10 candidate_status { last_received { term: 9 index: 84 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc"
I20250625 08:04:52.521656 3407 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 9 FOLLOWER]: Advancing to term 10
I20250625 08:04:52.521991 3544 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 10 candidate_status { last_received { term: 9 index: 84 } } ignore_live_leader: false dest_uuid: "755093bec4b54a9588598f814c3708e1" is_pre_election: true
I20250625 08:04:52.522251 3544 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f0b114bd68344b0bd2ee322bd5eccdc in current term 10: Already voted for candidate 755093bec4b54a9588598f814c3708e1 in this term.
W20250625 08:04:52.522393 3518 leader_election.cc:336] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:52.522696 3379 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:52.523634 3407 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755093bec4b54a9588598f814c3708e1 in term 10.
I20250625 08:04:52.523779 3600 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250625 08:04:52.523968 3521 leader_election.cc:304] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1; no voters: 0404561252194ce29bf2924fd5e67e22
I20250625 08:04:52.524168 3599 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 10 FOLLOWER]: Leader election won for term 10
I20250625 08:04:52.524369 3599 raft_consensus.cc:695] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [term 10 LEADER]: Becoming Leader. State: Replica: 755093bec4b54a9588598f814c3708e1, State: Running, Role: LEADER
I20250625 08:04:52.524670 3599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 9.84, Last appended by leader: 84, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:52.526250 3606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 10 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:52.526342 3606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:52.526615 3608 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:52.528264 3608 catalog_manager.cc:671] Loaded metadata for table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]
I20250625 08:04:52.528551 3608 catalog_manager.cc:671] Loaded metadata for table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]
I20250625 08:04:52.528770 3608 catalog_manager.cc:671] Loaded metadata for table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]
I20250625 08:04:52.528970 3608 catalog_manager.cc:671] Loaded metadata for table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]
I20250625 08:04:52.529163 3608 catalog_manager.cc:671] Loaded metadata for table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]
I20250625 08:04:52.529368 3608 catalog_manager.cc:671] Loaded metadata for table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]
I20250625 08:04:52.529568 3608 catalog_manager.cc:671] Loaded metadata for table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]
I20250625 08:04:52.529796 3608 catalog_manager.cc:671] Loaded metadata for table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]
I20250625 08:04:52.530975 3608 tablet_loader.cc:96] loaded metadata for tablet 0561d480e8b64f749c705741c3189f7f (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020])
I20250625 08:04:52.531841 3608 tablet_loader.cc:96] loaded metadata for tablet 39645fa894654da6949adbe9e3d1dc96 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875])
I20250625 08:04:52.532263 3608 tablet_loader.cc:96] loaded metadata for tablet 499fa735a33d450db3047beb0e018f6f (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a])
I20250625 08:04:52.532670 3608 tablet_loader.cc:96] loaded metadata for tablet 782a60aff92a4636bc98813f2214d1af (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206])
I20250625 08:04:52.532830 3608 tablet_loader.cc:96] loaded metadata for tablet 90154412b7b04b6db02f912a53000edb (table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904])
I20250625 08:04:52.533236 3608 tablet_loader.cc:96] loaded metadata for tablet a0db2bca14734321aa2167a9aade37e0 (table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d])
I20250625 08:04:52.533558 3608 tablet_loader.cc:96] loaded metadata for tablet b97c4e286a5040b1ad425242d9cb2dea (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6])
I20250625 08:04:52.533950 3608 tablet_loader.cc:96] loaded metadata for tablet cb5f968f985c422bbf615c31fd936ef1 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089])
I20250625 08:04:52.534691 3608 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:52.535369 3608 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:52.535418 3608 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:52.536149 3608 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:52.536772 3608 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Loaded TSK: 0
I20250625 08:04:52.537483 3608 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:52.538046 3608 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:52.538653 3608 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 20
I20250625 08:04:52.565735 3534 master_service.cc:948] ReplaceTablet: received request to replace tablet 782a60aff92a4636bc98813f2214d1af from {username='slave'} at 127.0.0.1:43504
I20250625 08:04:52.565846 3533 master_service.cc:948] ReplaceTablet: received request to replace tablet cb5f968f985c422bbf615c31fd936ef1 from {username='slave'} at 127.0.0.1:43504
I20250625 08:04:52.566418 3532 master_service.cc:948] ReplaceTablet: received request to replace tablet 39645fa894654da6949adbe9e3d1dc96 from {username='slave'} at 127.0.0.1:43504
I20250625 08:04:52.566371 3526 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43504:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:52.566622 3526 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43504:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:52.566692 3525 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43504:
name: "default.Table_f22b18eed5c34ffaa7af847a3c64598d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:52.566355 3528 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43504:
table { table_name: "default.Table_5eebaa7c05694491963fce48525e4681" } new_table_name: "default.Table_69a1c4361ab540a48fd24a5198251c2c" modify_external_catalogs: true
I20250625 08:04:52.567067 3527 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43504:
table { table_name: "default.Table_a3a8c500599e482da5b7120ec26a797a" } new_table_name: "default.Table_6894ca710ded4040ba337309d454d090" modify_external_catalogs: true
I20250625 08:04:52.567317 3529 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43504:
name: "default.Table_31023e55a4384ac88f8415871e52d142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:52.567497 3530 master_service.cc:948] ReplaceTablet: received request to replace tablet b97c4e286a5040b1ad425242d9cb2dea from {username='slave'} at 127.0.0.1:43504
I20250625 08:04:52.568123 3531 master_service.cc:948] ReplaceTablet: received request to replace tablet b97c4e286a5040b1ad425242d9cb2dea from {username='slave'} at 127.0.0.1:43504
I20250625 08:04:52.570662 3407 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Refusing update from remote peer 755093bec4b54a9588598f814c3708e1: Log matching property violated. Preceding OpId in replica: term: 9 index: 84. Preceding OpId from leader: term: 10 index: 86. (index mismatch)
W20250625 08:04:52.571019 3518 consensus_peers.cc:487] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 -> Peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Couldn't send request to peer 0404561252194ce29bf2924fd5e67e22. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:52.571264 3606 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
F20250625 08:04:52.572054 3588 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 4 out of 11 received events: NotificationEvent(eventId=21, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_31023e55a4384ac88f8415871e52d142, message={"server":"","servicePrincipal":"","db":"default","table":"table_31023e55a4384ac88f8415871e52d142","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_31023e55a4384ac88f8415871e52d142\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_31023e55a4384ac88f8415871e52d142\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_31023e55a4384ac88f8415871e52d142\",\"kudu.table_id\":\"7825ea67af2d42b68200097030b08116\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1750838669, eventType=ALTER_TABLE, dbName=default, tableName=table_7cb71508c2934e1aa091c71056ae7a01, message={"server":"","servicePrincipal":"","db":"default","table":"table_b49cbf7ee55042b39d7b5d3946a35618","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_b49cbf7ee55042b39d7b5d3946a35618\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_b49cbf7ee55042b39d7b5d3946a35618\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b49cbf7ee55042b39d7b5d3946a35618\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7cb71508c2934e1aa091c71056ae7a01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_7cb71508c2934e1aa091c71056ae7a01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7cb71508c2934e1aa091c71056ae7a01\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_d78071777f3646548f7f75cea674ecdc, message={"server":"","servicePrincipal":"","db":"default","table":"table_d78071777f3646548f7f75cea674ecdc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d78071777f3646548f7f75cea674ecdc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_d78071777f3646548f7f75cea674ecdc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d78071777f3646548f7f75cea674ecdc\",\"kudu.table_id\":\"0e43d85f40e545d9b31c5cb3e62f89a8\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=CREATE_TABLE, dbName=default, tableName=table_88079d64160a4318bf8ea1807cb3a30d, message={"server":"","servicePrincipal":"","db":"default","table":"table_88079d64160a4318bf8ea1807cb3a30d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_88079d64160a4318bf8ea1807cb3a30d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_88079d64160a4318bf8ea1807cb3a30d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_88079d64160a4318bf8ea1807cb3a30d\",\"kudu.table_id\":\"7c7e3cbdaf064697a864873ed87a21b6\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fad628afdcd google::LogMessage::Fail() at ??:0
I20250625 08:04:52.575456 3600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 755093bec4b54a9588598f814c3708e1. Latest consensus state: current_term: 10 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:52.575582 3600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fad628b3b93 google::LogMessage::SendToLog() at ??:0
I20250625 08:04:52.578038 3599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:52.578184 3599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1 [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:52.578534 3600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "755093bec4b54a9588598f814c3708e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:52.578644 3600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:52.578622 3610 mvcc.cc:204] Tried to move back new op lower bound from 7171435284768251904 to 7171435284583292928. Current Snapshot: MvccSnapshot[applied={T|T < 7171435284762382336}]
I20250625 08:04:52.580207 3609 mvcc.cc:204] Tried to move back new op lower bound from 7171435284768251904 to 7171435284583292928. Current Snapshot: MvccSnapshot[applied={T|T < 7171435284764725248}]
I20250625 08:04:52.580294 3530 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet b97c4e286a5040b1ad425242d9cb2dea
I20250625 08:04:52.581377 3533 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet cb5f968f985c422bbf615c31fd936ef1
I20250625 08:04:52.581522 3530 catalog_manager.cc:6652] ReplaceTablet: tablet b97c4e286a5040b1ad425242d9cb2dea deleted and replaced by tablet 03cc8f661a7f4245a6d1a601e9a5035c
I20250625 08:04:52.581637 3533 catalog_manager.cc:6652] ReplaceTablet: tablet cb5f968f985c422bbf615c31fd936ef1 deleted and replaced by tablet 4b5dc84827a0457983d91427688e0c67
I20250625 08:04:52.581815 3530 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43504:
table { table_name: "default.Table_88079d64160a4318bf8ea1807cb3a30d" } modify_external_catalogs: true
I20250625 08:04:52.581853 3533 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43504:
table { table_name: "default.Table_23c54834c68a47bd9654b34281b53e7d" } new_table_name: "default.Table_fbd9fab6fc2a480f88e12bb1ada0b9f6" modify_external_catalogs: true
I20250625 08:04:52.581974 3530 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43504:
table { table_name: "default.Table_88079d64160a4318bf8ea1807cb3a30d" } modify_external_catalogs: true
I20250625 08:04:52.582401 3531 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43504:
name: "default.Table_d78071777f3646548f7f75cea674ecdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:52.582501 3534 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet 782a60aff92a4636bc98813f2214d1af
I20250625 08:04:52.582718 3534 catalog_manager.cc:6652] ReplaceTablet: tablet 782a60aff92a4636bc98813f2214d1af deleted and replaced by tablet 85074a5f6dfb4c4b9c869a4141175da6
I20250625 08:04:52.582892 3534 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43504:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:52.583019 3534 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43504:
table { table_name: "default.Table_d28f89eb30604a7abf867f68131b2c1a" } modify_external_catalogs: true
I20250625 08:04:52.582727 3532 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 755093bec4b54a9588598f814c3708e1: Sending DeleteTablet for 3 replicas of tablet 39645fa894654da6949adbe9e3d1dc96
I20250625 08:04:52.583340 3532 catalog_manager.cc:6652] ReplaceTablet: tablet 39645fa894654da6949adbe9e3d1dc96 deleted and replaced by tablet db339d8e6f8343d994e0ba656d00acb2
I20250625 08:04:52.583515 3532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43504:
name: "default.Table_28db8798a1444ed2b1d46a5a3eba537b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fad628af7cc google::LogMessage::Flush() at ??:0
@ 0x7fad628afa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fad67848c91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 08:04:52.593910 1687 tablet_service.cc:1515] Processing DeleteTablet for tablet 39645fa894654da6949adbe9e3d1dc96 with delete_type TABLET_DATA_DELETED (replaced by tablet db339d8e6f8343d994e0ba656d00acb2) from {username='slave'} at 127.0.0.1:44964
I20250625 08:04:52.594064 1688 tablet_service.cc:1515] Processing DeleteTablet for tablet b97c4e286a5040b1ad425242d9cb2dea with delete_type TABLET_DATA_DELETED (replaced by tablet 03cc8f661a7f4245a6d1a601e9a5035c) from {username='slave'} at 127.0.0.1:44964
I20250625 08:04:52.594240 1691 tablet_service.cc:1515] Processing DeleteTablet for tablet cb5f968f985c422bbf615c31fd936ef1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5dc84827a0457983d91427688e0c67) from {username='slave'} at 127.0.0.1:44964
I20250625 08:04:52.594359 1689 tablet_service.cc:1515] Processing DeleteTablet for tablet 782a60aff92a4636bc98813f2214d1af with delete_type TABLET_DATA_DELETED (replaced by tablet 85074a5f6dfb4c4b9c869a4141175da6) from {username='slave'} at 127.0.0.1:44964
I20250625 08:04:52.594681 1686 tablet_service.cc:1468] Processing CreateTablet for tablet 4b5dc84827a0457983d91427688e0c67 (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.595043 1686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b5dc84827a0457983d91427688e0c67. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fad67847ba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 08:04:52.597191 1825 tablet_service.cc:1515] Processing DeleteTablet for tablet cb5f968f985c422bbf615c31fd936ef1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5dc84827a0457983d91427688e0c67) from {username='slave'} at 127.0.0.1:33624
I20250625 08:04:52.597499 1827 tablet_service.cc:1515] Processing DeleteTablet for tablet 782a60aff92a4636bc98813f2214d1af with delete_type TABLET_DATA_DELETED (replaced by tablet 85074a5f6dfb4c4b9c869a4141175da6) from {username='slave'} at 127.0.0.1:33624
I20250625 08:04:52.597617 1824 tablet_service.cc:1515] Processing DeleteTablet for tablet 39645fa894654da6949adbe9e3d1dc96 with delete_type TABLET_DATA_DELETED (replaced by tablet db339d8e6f8343d994e0ba656d00acb2) from {username='slave'} at 127.0.0.1:33624
I20250625 08:04:52.597623 1685 tablet_service.cc:1468] Processing CreateTablet for tablet 03cc8f661a7f4245a6d1a601e9a5035c (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.597880 1685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03cc8f661a7f4245a6d1a601e9a5035c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:52.597892 1828 tablet_service.cc:1468] Processing CreateTablet for tablet 4b5dc84827a0457983d91427688e0c67 (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.598196 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b5dc84827a0457983d91427688e0c67. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fad6784730b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 08:04:52.600354 1823 tablet_service.cc:1515] Processing DeleteTablet for tablet b97c4e286a5040b1ad425242d9cb2dea with delete_type TABLET_DATA_DELETED (replaced by tablet 03cc8f661a7f4245a6d1a601e9a5035c) from {username='slave'} at 127.0.0.1:33624
I20250625 08:04:52.601111 1826 tablet_service.cc:1468] Processing CreateTablet for tablet 03cc8f661a7f4245a6d1a601e9a5035c (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.601284 3630 tablet_bootstrap.cc:492] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:52.601401 1826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03cc8f661a7f4245a6d1a601e9a5035c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fad6784ca12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20250625 08:04:52.602389 3630 tablet_bootstrap.cc:654] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.602531 1961 tablet_service.cc:1515] Processing DeleteTablet for tablet b97c4e286a5040b1ad425242d9cb2dea with delete_type TABLET_DATA_DELETED (replaced by tablet 03cc8f661a7f4245a6d1a601e9a5035c) from {username='slave'} at 127.0.0.1:37372
I20250625 08:04:52.603111 3629 tablet_replica.cc:331] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:52.603193 1963 tablet_service.cc:1515] Processing DeleteTablet for tablet 782a60aff92a4636bc98813f2214d1af with delete_type TABLET_DATA_DELETED (replaced by tablet 85074a5f6dfb4c4b9c869a4141175da6) from {username='slave'} at 127.0.0.1:37372
I20250625 08:04:52.603258 3629 raft_consensus.cc:2241] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:52.603327 1959 tablet_service.cc:1515] Processing DeleteTablet for tablet 39645fa894654da6949adbe9e3d1dc96 with delete_type TABLET_DATA_DELETED (replaced by tablet db339d8e6f8343d994e0ba656d00acb2) from {username='slave'} at 127.0.0.1:37372
I20250625 08:04:52.603391 1962 tablet_service.cc:1468] Processing CreateTablet for tablet 03cc8f661a7f4245a6d1a601e9a5035c (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.603477 3629 raft_consensus.cc:2270] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.603554 1960 tablet_service.cc:1468] Processing CreateTablet for tablet 4b5dc84827a0457983d91427688e0c67 (DEFAULT_TABLE table=default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.603655 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03cc8f661a7f4245a6d1a601e9a5035c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:52.603763 1960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b5dc84827a0457983d91427688e0c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:52.603868 3629 ts_tablet_manager.cc:1905] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.601125 3631 tablet_replica.cc:331] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:52.605794 1961 tablet_service.cc:1515] Processing DeleteTablet for tablet cb5f968f985c422bbf615c31fd936ef1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b5dc84827a0457983d91427688e0c67) from {username='slave'} at 127.0.0.1:37372
I20250625 08:04:52.605882 3631 raft_consensus.cc:2241] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:52.605974 3629 ts_tablet_manager.cc:1918] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.606040 3629 log.cc:1199] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/39645fa894654da6949adbe9e3d1dc96
I20250625 08:04:52.606035 3631 raft_consensus.cc:2270] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.606355 3629 ts_tablet_manager.cc:1939] T 39645fa894654da6949adbe9e3d1dc96 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:52.606720 1826 tablet_service.cc:1468] Processing CreateTablet for tablet db339d8e6f8343d994e0ba656d00acb2 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.606971 1826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db339d8e6f8343d994e0ba656d00acb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:52.606981 3629 tablet_replica.cc:331] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:52.607023 1961 tablet_service.cc:1468] Processing CreateTablet for tablet db339d8e6f8343d994e0ba656d00acb2 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.607096 3629 raft_consensus.cc:2241] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:52.607251 1961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db339d8e6f8343d994e0ba656d00acb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:52.607285 3629 raft_consensus.cc:2270] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.607569 1828 tablet_service.cc:1468] Processing CreateTablet for tablet 85074a5f6dfb4c4b9c869a4141175da6 (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.607632 3629 ts_tablet_manager.cc:1905] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.608745 1828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85074a5f6dfb4c4b9c869a4141175da6. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x55a0975da752 std::function<>::operator()() at ??:0
I20250625 08:04:52.609540 3629 ts_tablet_manager.cc:1918] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.609604 3629 log.cc:1199] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/b97c4e286a5040b1ad425242d9cb2dea
I20250625 08:04:52.609760 1685 tablet_service.cc:1468] Processing CreateTablet for tablet db339d8e6f8343d994e0ba656d00acb2 (DEFAULT_TABLE table=default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.609848 3629 ts_tablet_manager.cc:1939] T b97c4e286a5040b1ad425242d9cb2dea P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:52.609987 1685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db339d8e6f8343d994e0ba656d00acb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:52.610477 3629 tablet_replica.cc:331] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:52.610579 3629 raft_consensus.cc:2241] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:52.610622 1686 tablet_service.cc:1468] Processing CreateTablet for tablet 85074a5f6dfb4c4b9c869a4141175da6 (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.610677 3629 raft_consensus.cc:2270] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.610819 1686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85074a5f6dfb4c4b9c869a4141175da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:52.611011 3629 ts_tablet_manager.cc:1905] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.611090 3633 tablet_bootstrap.cc:492] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:52.612107 3633 tablet_bootstrap.cc:654] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.612854 3636 tablet_bootstrap.cc:492] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:52.613214 3631 ts_tablet_manager.cc:1905] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.613319 3629 ts_tablet_manager.cc:1918] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.613384 3629 log.cc:1199] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/cb5f968f985c422bbf615c31fd936ef1
I20250625 08:04:52.613623 3629 ts_tablet_manager.cc:1939] T cb5f968f985c422bbf615c31fd936ef1 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:52.614130 3629 tablet_replica.cc:331] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:52.614151 3636 tablet_bootstrap.cc:654] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.614260 3629 raft_consensus.cc:2241] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:52.614365 3629 raft_consensus.cc:2270] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.614732 3629 ts_tablet_manager.cc:1905] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.614831 3521 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 39645fa894654da6949adbe9e3d1dc96 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:52.615005 3521 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet b97c4e286a5040b1ad425242d9cb2dea (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]) successfully deleted
I20250625 08:04:52.615144 3521 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet cb5f968f985c422bbf615c31fd936ef1 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:52.607019 1962 tablet_service.cc:1468] Processing CreateTablet for tablet 85074a5f6dfb4c4b9c869a4141175da6 (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:52.615727 1962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85074a5f6dfb4c4b9c869a4141175da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:52.615748 3631 ts_tablet_manager.cc:1918] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.615823 3631 log.cc:1199] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/cb5f968f985c422bbf615c31fd936ef1
I20250625 08:04:52.616171 3631 ts_tablet_manager.cc:1939] T cb5f968f985c422bbf615c31fd936ef1 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:52.617015 3631 tablet_replica.cc:331] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:52.617049 3629 ts_tablet_manager.cc:1918] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.617117 3629 log.cc:1199] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/782a60aff92a4636bc98813f2214d1af
I20250625 08:04:52.617161 3631 raft_consensus.cc:2241] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:52.617254 3631 raft_consensus.cc:2270] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.617372 3629 ts_tablet_manager.cc:1939] T 782a60aff92a4636bc98813f2214d1af P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:52.617594 3631 ts_tablet_manager.cc:1905] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.617986 3521 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet cb5f968f985c422bbf615c31fd936ef1 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]) successfully deleted
I20250625 08:04:52.618144 3521 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 782a60aff92a4636bc98813f2214d1af (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]) successfully deleted
I20250625 08:04:52.618690 3634 tablet_replica.cc:331] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:52.618788 3634 raft_consensus.cc:2241] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:52.618893 3634 raft_consensus.cc:2270] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fad63d010b7 kudu::Thread::SuperviseThread() at ??:0
I20250625 08:04:52.619694 3631 ts_tablet_manager.cc:1918] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.619841 3631 log.cc:1199] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/782a60aff92a4636bc98813f2214d1af
I20250625 08:04:52.620312 3634 ts_tablet_manager.cc:1905] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.621469 3631 ts_tablet_manager.cc:1939] T 782a60aff92a4636bc98813f2214d1af P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:52.621698 3630 tablet_bootstrap.cc:492] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:52.621789 3630 ts_tablet_manager.cc:1397] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20250625 08:04:52.622113 3521 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 782a60aff92a4636bc98813f2214d1af (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]) successfully deleted
I20250625 08:04:52.622371 3631 tablet_replica.cc:331] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:52.622493 3631 raft_consensus.cc:2241] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:52.622586 3631 raft_consensus.cc:2270] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.622557 3630 raft_consensus.cc:357] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.622687 3630 raft_consensus.cc:383] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.622723 3630 raft_consensus.cc:738] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.622756 3634 ts_tablet_manager.cc:1918] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.622818 3634 log.cc:1199] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/b97c4e286a5040b1ad425242d9cb2dea
I20250625 08:04:52.622895 3631 ts_tablet_manager.cc:1905] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.622864 3630 consensus_queue.cc:260] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.623106 3634 ts_tablet_manager.cc:1939] T b97c4e286a5040b1ad425242d9cb2dea P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:52.623205 3630 ts_tablet_manager.cc:1428] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 08:04:52.623400 3630 tablet_bootstrap.cc:492] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:52.623698 3634 tablet_replica.cc:331] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:52.623813 3634 raft_consensus.cc:2241] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:52.624004 3634 raft_consensus.cc:2270] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.624361 3634 ts_tablet_manager.cc:1905] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.624441 3630 tablet_bootstrap.cc:654] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.624861 3631 ts_tablet_manager.cc:1918] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.624918 3631 log.cc:1199] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/39645fa894654da6949adbe9e3d1dc96
I20250625 08:04:52.625149 3631 ts_tablet_manager.cc:1939] T 39645fa894654da6949adbe9e3d1dc96 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:52.625823 3521 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 39645fa894654da6949adbe9e3d1dc96 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]) successfully deleted
I20250625 08:04:52.626052 3631 tablet_replica.cc:331] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:52.626148 3631 raft_consensus.cc:2241] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:52.626233 3631 raft_consensus.cc:2270] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.626529 3634 ts_tablet_manager.cc:1918] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.626581 3631 ts_tablet_manager.cc:1905] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.626595 3634 log.cc:1199] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/782a60aff92a4636bc98813f2214d1af
I20250625 08:04:52.626842 3634 ts_tablet_manager.cc:1939] T 782a60aff92a4636bc98813f2214d1af P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:52.626871 3633 tablet_bootstrap.cc:492] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:52.626945 3633 ts_tablet_manager.cc:1397] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250625 08:04:52.627058 3521 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet b97c4e286a5040b1ad425242d9cb2dea (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]) successfully deleted
I20250625 08:04:52.627410 3634 tablet_replica.cc:331] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:52.627408 3633 raft_consensus.cc:357] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.627480 3521 catalog_manager.cc:4928] TS 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): tablet 782a60aff92a4636bc98813f2214d1af (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]) successfully deleted
I20250625 08:04:52.627521 3633 raft_consensus.cc:383] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.627521 3634 raft_consensus.cc:2241] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:52.627553 3633 raft_consensus.cc:738] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.627624 3634 raft_consensus.cc:2270] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.628034 3634 ts_tablet_manager.cc:1905] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.628417 3636 tablet_bootstrap.cc:492] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:52.628495 3636 ts_tablet_manager.cc:1397] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
@ 0x7fad644f06db start_thread at ??:0
I20250625 08:04:52.628999 3636 raft_consensus.cc:357] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.629109 3636 raft_consensus.cc:383] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.629113 3631 ts_tablet_manager.cc:1918] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.629148 3636 raft_consensus.cc:738] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.629172 3631 log.cc:1199] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/b97c4e286a5040b1ad425242d9cb2dea
I20250625 08:04:52.629429 3631 ts_tablet_manager.cc:1939] T b97c4e286a5040b1ad425242d9cb2dea P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:52.629818 3633 consensus_queue.cc:260] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.630179 3521 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet b97c4e286a5040b1ad425242d9cb2dea (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]) successfully deleted
I20250625 08:04:52.630486 3634 ts_tablet_manager.cc:1918] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.630558 3634 log.cc:1199] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/39645fa894654da6949adbe9e3d1dc96
I20250625 08:04:52.630828 3634 ts_tablet_manager.cc:1939] T 39645fa894654da6949adbe9e3d1dc96 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
@ 0x7fad61a6671f clone at ??:0
I20250625 08:04:52.631470 3634 tablet_replica.cc:331] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:52.631604 3634 raft_consensus.cc:2241] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:52.631786 3634 raft_consensus.cc:2270] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:52.632135 3634 ts_tablet_manager.cc:1905] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:52.632619 3633 ts_tablet_manager.cc:1428] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250625 08:04:52.632807 3633 tablet_bootstrap.cc:492] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:52.633885 3633 tablet_bootstrap.cc:654] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.633970 3634 ts_tablet_manager.cc:1918] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:52.634035 3634 log.cc:1199] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/cb5f968f985c422bbf615c31fd936ef1
I20250625 08:04:52.634295 3634 ts_tablet_manager.cc:1939] T cb5f968f985c422bbf615c31fd936ef1 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:52.634575 3630 tablet_bootstrap.cc:492] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:52.634683 3630 ts_tablet_manager.cc:1397] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250625 08:04:52.635182 3630 raft_consensus.cc:357] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.635282 3630 raft_consensus.cc:383] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.635308 3630 raft_consensus.cc:738] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.635423 3630 consensus_queue.cc:260] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.635653 3630 ts_tablet_manager.cc:1428] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:52.635816 3630 tablet_bootstrap.cc:492] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:52.636811 3630 tablet_bootstrap.cc:654] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.638069 3636 consensus_queue.cc:260] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.638423 3636 ts_tablet_manager.cc:1428] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250625 08:04:52.638685 3636 tablet_bootstrap.cc:492] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:52.640015 3636 tablet_bootstrap.cc:654] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.641549 3630 tablet_bootstrap.cc:492] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:52.641641 3630 ts_tablet_manager.cc:1397] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 08:04:52.641958 3633 tablet_bootstrap.cc:492] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:52.642050 3633 ts_tablet_manager.cc:1397] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 08:04:52.642200 3630 raft_consensus.cc:357] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.642261 3636 tablet_bootstrap.cc:492] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:52.642297 3630 raft_consensus.cc:383] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.642329 3630 raft_consensus.cc:738] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.642359 3636 ts_tablet_manager.cc:1397] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250625 08:04:52.642483 3630 consensus_queue.cc:260] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.642653 3633 raft_consensus.cc:357] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.642751 3633 raft_consensus.cc:383] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.642768 3630 ts_tablet_manager.cc:1428] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:52.642788 3633 raft_consensus.cc:738] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.642982 3630 tablet_bootstrap.cc:492] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:52.642899 3636 raft_consensus.cc:357] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.643028 3636 raft_consensus.cc:383] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.643059 3636 raft_consensus.cc:738] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.643195 3636 consensus_queue.cc:260] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.643327 3633 consensus_queue.cc:260] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.643523 3636 ts_tablet_manager.cc:1428] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 08:04:52.643637 3633 ts_tablet_manager.cc:1428] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 08:04:52.643730 3636 tablet_bootstrap.cc:492] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:52.643821 3633 tablet_bootstrap.cc:492] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:52.644021 3630 tablet_bootstrap.cc:654] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.644902 3633 tablet_bootstrap.cc:654] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.644979 3636 tablet_bootstrap.cc:654] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.646318 3633 tablet_bootstrap.cc:492] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:52.646380 3633 ts_tablet_manager.cc:1397] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:52.646874 3633 raft_consensus.cc:357] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.646984 3633 raft_consensus.cc:383] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.647019 3633 raft_consensus.cc:738] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.647138 3633 consensus_queue.cc:260] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.647420 3633 ts_tablet_manager.cc:1428] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:52.647598 3633 tablet_bootstrap.cc:492] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:52.648231 3630 tablet_bootstrap.cc:492] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:52.648311 3630 ts_tablet_manager.cc:1397] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 08:04:52.648627 3633 tablet_bootstrap.cc:654] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.648795 3630 raft_consensus.cc:357] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.648892 3630 raft_consensus.cc:383] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.648932 3630 raft_consensus.cc:738] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.649044 3630 consensus_queue.cc:260] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.649338 3630 ts_tablet_manager.cc:1428] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:52.650646 3633 tablet_bootstrap.cc:492] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:52.650727 3633 ts_tablet_manager.cc:1397] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:52.651228 3633 raft_consensus.cc:357] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.651335 3633 raft_consensus.cc:383] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.651371 3633 raft_consensus.cc:738] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.651520 3633 consensus_queue.cc:260] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.651660 3636 tablet_bootstrap.cc:492] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:52.651753 3636 ts_tablet_manager.cc:1397] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20250625 08:04:52.651795 3633 ts_tablet_manager.cc:1428] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:52.652383 3636 raft_consensus.cc:357] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.652491 3636 raft_consensus.cc:383] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.652521 3636 raft_consensus.cc:738] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.652669 3636 consensus_queue.cc:260] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.653147 3636 ts_tablet_manager.cc:1428] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:52.653333 3636 tablet_bootstrap.cc:492] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:52.654803 3636 tablet_bootstrap.cc:654] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:52.655984 3636 tablet_bootstrap.cc:492] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:52.656069 3636 ts_tablet_manager.cc:1397] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:52.656515 3636 raft_consensus.cc:357] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.656616 3636 raft_consensus.cc:383] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:52.656651 3636 raft_consensus.cc:738] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:52.656785 3636 consensus_queue.cc:260] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.657096 3636 ts_tablet_manager.cc:1428] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:52.754606 3336 raft_consensus.cc:491] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:52.754732 3336 raft_consensus.cc:513] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.755234 3336 leader_election.cc:290] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:52.755277 3657 raft_consensus.cc:491] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:52.755348 3657 raft_consensus.cc:513] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.755667 3657 leader_election.cc:290] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:52.755779 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cc8f661a7f4245a6d1a601e9a5035c" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:52.755774 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cc8f661a7f4245a6d1a601e9a5035c" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:52.755897 1711 raft_consensus.cc:2466] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:52.755937 1847 raft_consensus.cc:2466] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:52.756393 1919 leader_election.cc:304] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:52.756533 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b5dc84827a0457983d91427688e0c67" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:52.756537 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b5dc84827a0457983d91427688e0c67" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:52.756637 1711 raft_consensus.cc:2466] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:52.756661 3657 raft_consensus.cc:2802] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:52.756670 1847 raft_consensus.cc:2466] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:52.756707 3657 raft_consensus.cc:491] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:52.756747 3657 raft_consensus.cc:3058] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:52.756918 1919 leader_election.cc:304] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:52.757169 3336 raft_consensus.cc:2802] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:52.757236 3336 raft_consensus.cc:491] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:52.757277 3336 raft_consensus.cc:3058] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:52.757925 3657 raft_consensus.cc:513] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.758198 3336 raft_consensus.cc:513] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.758311 3657 leader_election.cc:290] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:52.758667 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cc8f661a7f4245a6d1a601e9a5035c" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:52.758771 3336 leader_election.cc:290] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:52.758769 1711 raft_consensus.cc:3058] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:52.758975 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cc8f661a7f4245a6d1a601e9a5035c" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:52.759074 1847 raft_consensus.cc:3058] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:52.759093 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b5dc84827a0457983d91427688e0c67" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:52.759218 1848 raft_consensus.cc:3058] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:52.759754 1711 raft_consensus.cc:2466] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:52.760044 1847 raft_consensus.cc:2466] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:52.760260 1848 raft_consensus.cc:2466] T 4b5dc84827a0457983d91427688e0c67 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:52.760349 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b5dc84827a0457983d91427688e0c67" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:52.760349 1919 leader_election.cc:304] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:52.760445 1711 raft_consensus.cc:3058] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:52.760643 1919 leader_election.cc:304] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:52.760844 3657 raft_consensus.cc:2802] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:52.760866 3336 raft_consensus.cc:2802] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:52.761082 3336 raft_consensus.cc:695] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:52.761092 3657 raft_consensus.cc:695] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:52.761247 3336 consensus_queue.cc:237] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.761368 1711 raft_consensus.cc:2466] T 4b5dc84827a0457983d91427688e0c67 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:52.761680 3657 consensus_queue.cc:237] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.769774 2930 raft_consensus.cc:491] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:52.769902 2930 raft_consensus.cc:513] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.770372 2930 leader_election.cc:290] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:52.770792 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85074a5f6dfb4c4b9c869a4141175da6" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:52.770880 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85074a5f6dfb4c4b9c869a4141175da6" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:52.770924 1983 raft_consensus.cc:2466] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:52.771049 1711 raft_consensus.cc:2466] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:52.771267 1784 leader_election.cc:304] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:52.771428 2930 raft_consensus.cc:2802] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:52.771486 2930 raft_consensus.cc:491] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:52.771528 2930 raft_consensus.cc:3058] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:52.772389 2930 raft_consensus.cc:513] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:52.772728 2930 leader_election.cc:290] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:52.773133 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85074a5f6dfb4c4b9c869a4141175da6" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:52.773155 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85074a5f6dfb4c4b9c869a4141175da6" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:52.773268 1711 raft_consensus.cc:3058] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:52.773267 1983 raft_consensus.cc:3058] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:52.774008 1711 raft_consensus.cc:2466] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:52.774278 1784 leader_election.cc:304] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:52.774338 1983 raft_consensus.cc:2466] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:52.774467 2930 raft_consensus.cc:2802] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:52.774622 2930 raft_consensus.cc:695] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:52.774783 2930 consensus_queue.cc:237] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
W20250625 08:04:52.804553 1917 connection.cc:537] server connection from 127.0.0.1:37372 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:52.804750 1782 connection.cc:537] client connection to 127.1.19.124:37745 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:52.805044 2037 connection.cc:537] client connection to 127.1.19.124:37745 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:52.805495 2037 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.19.124:37745)
W20250625 08:04:52.805820 2037 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.124:37745)
W20250625 08:04:52.806022 2037 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.19.124:37745)
W20250625 08:04:52.806118 2037 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.19.124:37745)
W20250625 08:04:52.806679 3503 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:52.805312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.124:37745 (local address 127.1.19.66:41591)
0625 08:04:52.805550 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:52.805566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:52.805660 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:52.806557 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145}
W20250625 08:04:52.807165 3594 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 08:04:52.805796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.19.124:37745 (local address 127.1.19.67:41911)
0625 08:04:52.805975 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0625 08:04:52.805990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0625 08:04:52.806065 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 08:04:52.807040 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":100}
I20250625 08:04:53.068993 3494 raft_consensus.cc:491] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:53.069137 3494 raft_consensus.cc:513] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:53.069574 3494 leader_election.cc:290] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:53.070133 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db339d8e6f8343d994e0ba656d00acb2" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:53.070133 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db339d8e6f8343d994e0ba656d00acb2" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:53.070294 1983 raft_consensus.cc:2466] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:53.070289 1848 raft_consensus.cc:2466] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 0.
I20250625 08:04:53.070662 1647 leader_election.cc:304] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:53.070838 3494 raft_consensus.cc:2802] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:53.070896 3494 raft_consensus.cc:491] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:53.070926 3494 raft_consensus.cc:3058] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:53.071799 3494 raft_consensus.cc:513] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:53.072147 3494 leader_election.cc:290] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:53.072561 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db339d8e6f8343d994e0ba656d00acb2" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:53.072595 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db339d8e6f8343d994e0ba656d00acb2" candidate_uuid: "eebfa8472fe04080ac012e2c91bedaae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:53.072696 1848 raft_consensus.cc:3058] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:53.072696 1983 raft_consensus.cc:3058] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:53.073632 1848 raft_consensus.cc:2466] T db339d8e6f8343d994e0ba656d00acb2 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:53.073632 1983 raft_consensus.cc:2466] T db339d8e6f8343d994e0ba656d00acb2 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eebfa8472fe04080ac012e2c91bedaae in term 1.
I20250625 08:04:53.073942 1647 leader_election.cc:304] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:53.074159 3494 raft_consensus.cc:2802] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:53.074265 3494 raft_consensus.cc:695] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [term 1 LEADER]: Becoming Leader. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Running, Role: LEADER
I20250625 08:04:53.074410 3494 consensus_queue.cc:237] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:53.098580 3667 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:53.098719 3667 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:53.099129 3667 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:53.100252 3379 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:53.100453 3378 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:53.100521 3378 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:53.100646 3667 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250625 08:04:53.146634 3336 consensus_queue.cc:1035] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:53.152283 3659 consensus_queue.cc:1035] T 4b5dc84827a0457983d91427688e0c67 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:53.207377 3662 consensus_queue.cc:1035] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:53.210934 3662 consensus_queue.cc:1035] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:53.341209 3659 consensus_queue.cc:1035] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:53.345041 3659 consensus_queue.cc:1035] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:53.571736 3481 consensus_queue.cc:1035] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:53.575989 3481 consensus_queue.cc:1035] T db339d8e6f8343d994e0ba656d00acb2 P eebfa8472fe04080ac012e2c91bedaae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:53.686897 3687 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:53.687026 3687 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:53.687458 3687 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:53.688660 3379 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:53.688742 3378 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:53.688800 3378 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:53.688958 3687 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250625 08:04:53.704830 3363 master_runner.cc:186] Error getting master registration for 127.1.19.125:38317: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111),
I20250625 08:04:53.704939 3363 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:54.424398 3699 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:54.424530 3699 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:54.424933 3699 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:54.425468 3378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20250625 08:04:54.426080 3379 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:04:54.426174 3378 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317): Network error: Client connection negotiation failed: client connection to 127.1.19.125:38317: connect: Connection refused (error 111)
I20250625 08:04:54.426252 3378 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 0404561252194ce29bf2924fd5e67e22, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:54.426389 3699 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250625 08:04:55.109694 1101 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
/tmp/dist-test-taskYJaGNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.125:38317
--webserver_interface=127.1.19.125
--webserver_port=39497
--builtin_ntp_servers=127.1.19.84:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 08:04:55.221799 3709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 08:04:55.222148 3709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 08:04:55.222213 3709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 08:04:55.226727 3709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 08:04:55.226835 3709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 08:04:55.226869 3709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 08:04:55.226907 3709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 08:04:55.230945 3709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.84:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:34703
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.125:38317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.19.125
--webserver_port=39497
--never_fsync=true
--heap_profile_path=/tmp/kudu.3709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type DEBUG
built by None at 25 Jun 2025 07:43:14 UTC on 24a791456cd2
build id 6758
I20250625 08:04:55.232079 3709 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 08:04:55.233183 3709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 08:04:55.240532 3715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:55.240583 3718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 08:04:55.240597 3716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 08:04:55.240897 3709 server_base.cc:1048] running on GCE node
I20250625 08:04:55.241374 3709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 08:04:55.242267 3709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 08:04:55.243427 3709 hybrid_clock.cc:648] HybridClock initialized: now 1750838695243411 us; error 32 us; skew 500 ppm
I20250625 08:04:55.245379 3709 webserver.cc:469] Webserver started at http://127.1.19.125:39497/ using document root <none> and password file <none>
I20250625 08:04:55.245960 3709 fs_manager.cc:362] Metadata directory not provided
I20250625 08:04:55.246019 3709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 08:04:55.249444 3709 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:55.251402 3724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 08:04:55.252426 3709 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250625 08:04:55.252532 3709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data,/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
uuid: "0404561252194ce29bf2924fd5e67e22"
format_stamp: "Formatted at 2025-06-25 08:04:26 on dist-test-slave-4qtp"
I20250625 08:04:55.252936 3709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 08:04:55.268630 3709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 08:04:55.269320 3709 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 08:04:55.269492 3709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 08:04:55.277130 3709 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.125:38317
I20250625 08:04:55.277135 3776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.125:38317 every 8 connection(s)
I20250625 08:04:55.278474 3709 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/master-1/data/info.pb
I20250625 08:04:55.283911 3778 sys_catalog.cc:263] Verifying existing consensus state
I20250625 08:04:55.285985 3778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap starting.
I20250625 08:04:55.286680 1101 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskYJaGNk/build/debug/bin/kudu as pid 3709
I20250625 08:04:55.291844 2031 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:55.292719 1896 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:55.298947 3778 log.cc:826] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Log is configured to *not* fsync() on all Append() calls
I20250625 08:04:55.335004 3778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap replayed 1/1 log segments. Stats: ops{read=74 overwritten=1 applied=73 ignored=0} inserts{seen=47 ignored=0} mutations{seen=60 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:04:55.335460 3778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: Bootstrap complete.
I20250625 08:04:55.339208 3778 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:55.339656 3778 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0404561252194ce29bf2924fd5e67e22, State: Initialized, Role: FOLLOWER
I20250625 08:04:55.340186 3778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 73, Last appended: 7.73, Last appended by leader: 73, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:55.340795 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:55.340937 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:55.341477 3778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 08:04:55.350385 3801 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:55.350484 3801 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: loading cluster ID for follower catalog manager: success
I20250625 08:04:55.353443 3801 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: acquiring CA information for follower catalog manager: success
I20250625 08:04:55.355305 3801 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 08:04:55.360529 3709 master_runner.cc:186] Error getting master registration for 127.1.19.124:37745: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111),
I20250625 08:04:55.362044 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:04:55.454105 1759 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:55.454267 1759 heartbeater.cc:499] Master 127.1.19.125:38317 was elected leader, sending a full tablet report...
I20250625 08:04:55.455731 3741 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" instance_seqno: 1750838666815429) as {username='slave'} at 127.1.19.65:36425; Asking this server to re-register.
I20250625 08:04:55.456238 1759 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:55.457453 3741 ts_manager.cc:194] Registered new tserver with Master: eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
W20250625 08:04:55.692845 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 21: this message will repeat every 5th retry.
I20250625 08:04:55.724710 3787 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:55.724960 3787 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:55.726217 3787 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:55.726729 3407 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0404561252194ce29bf2924fd5e67e22" candidate_term: 8 candidate_status { last_received { term: 7 index: 73 } } ignore_live_leader: false dest_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" is_pre_election: true
W20250625 08:04:55.726801 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:55.726933 3407 raft_consensus.cc:2366] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0404561252194ce29bf2924fd5e67e22 for earlier term 8. Current term is 10.
I20250625 08:04:55.727382 3728 leader_election.cc:400] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0404561252194ce29bf2924fd5e67e22 for earlier term 8. Current term is 10.
I20250625 08:04:55.727438 3728 leader_election.cc:403] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20250625 08:04:55.727628 3787 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 7 FOLLOWER]: Advancing to term 10
W20250625 08:04:55.727628 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:55.729418 3787 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 10 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0404561252194ce29bf2924fd5e67e22 for earlier term 8. Current term is 10.
I20250625 08:04:55.795568 2031 heartbeater.cc:344] Connected to a master server at 127.1.19.125:38317
I20250625 08:04:55.795801 3741 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" instance_seqno: 1750838667017582) as {username='slave'} at 127.1.19.66:54257; Asking this server to re-register.
I20250625 08:04:55.796204 1896 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:55.796768 3741 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" instance_seqno: 1750838667205184) as {username='slave'} at 127.1.19.67:46551; Asking this server to re-register.
I20250625 08:04:55.797138 3740 ts_manager.cc:194] Registered new tserver with Master: de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
I20250625 08:04:55.797174 2031 heartbeater.cc:461] Registering TS with master...
I20250625 08:04:55.797976 3740 ts_manager.cc:194] Registered new tserver with Master: 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
I20250625 08:04:55.935467 3808 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:55.935650 3808 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:55.936216 3808 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:55.937747 3379 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:55.940665 3751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 11 candidate_status { last_received { term: 10 index: 91 } } ignore_live_leader: false dest_uuid: "0404561252194ce29bf2924fd5e67e22" is_pre_election: true
I20250625 08:04:55.941002 3751 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f0b114bd68344b0bd2ee322bd5eccdc in term 10.
I20250625 08:04:55.941597 3378 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 755093bec4b54a9588598f814c3708e1
I20250625 08:04:55.941807 3808 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250625 08:04:55.941876 3808 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Starting leader election (detected failure of leader 755093bec4b54a9588598f814c3708e1)
I20250625 08:04:55.941973 3808 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 10 FOLLOWER]: Advancing to term 11
W20250625 08:04:55.942934 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 21: this message will repeat every 5th retry.
I20250625 08:04:55.943572 3808 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:55.944046 3808 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 election: Requested vote from peers 0404561252194ce29bf2924fd5e67e22 (127.1.19.125:38317), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
I20250625 08:04:55.944958 3751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" candidate_term: 11 candidate_status { last_received { term: 10 index: 91 } } ignore_live_leader: false dest_uuid: "0404561252194ce29bf2924fd5e67e22"
I20250625 08:04:55.945104 3751 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 10 FOLLOWER]: Advancing to term 11
W20250625 08:04:55.945544 3379 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:55.946233 3751 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f0b114bd68344b0bd2ee322bd5eccdc in term 11.
I20250625 08:04:55.946573 3378 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22, 5f0b114bd68344b0bd2ee322bd5eccdc; no voters: 755093bec4b54a9588598f814c3708e1
I20250625 08:04:55.946745 3808 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 11 FOLLOWER]: Leader election won for term 11
I20250625 08:04:55.947016 3808 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [term 11 LEADER]: Becoming Leader. State: Replica: 5f0b114bd68344b0bd2ee322bd5eccdc, State: Running, Role: LEADER
I20250625 08:04:55.947376 3808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 10.91, Last appended by leader: 91, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:55.948408 3812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0b114bd68344b0bd2ee322bd5eccdc. Latest consensus state: current_term: 11 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:55.948529 3812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: LEADER
I20250625 08:04:55.948832 3814 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 08:04:55.950655 3814 catalog_manager.cc:671] Loaded metadata for table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020]
I20250625 08:04:55.950981 3814 catalog_manager.cc:671] Loaded metadata for table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875]
I20250625 08:04:55.951216 3814 catalog_manager.cc:671] Loaded metadata for table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]
I20250625 08:04:55.951455 3814 catalog_manager.cc:671] Loaded metadata for table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]
I20250625 08:04:55.951663 3814 catalog_manager.cc:671] Loaded metadata for table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904]
I20250625 08:04:55.951867 3814 catalog_manager.cc:671] Loaded metadata for table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]
I20250625 08:04:55.952101 3814 catalog_manager.cc:671] Loaded metadata for table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a]
I20250625 08:04:55.952342 3814 catalog_manager.cc:671] Loaded metadata for table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089]
I20250625 08:04:55.953584 3814 tablet_loader.cc:96] loaded metadata for tablet 03cc8f661a7f4245a6d1a601e9a5035c (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6])
I20250625 08:04:55.953812 3814 tablet_loader.cc:96] loaded metadata for tablet 0561d480e8b64f749c705741c3189f7f (table default.table_a3a8c500599e482da5b7120ec26a797a [id=06139221a5c54ced9f9903b3c0c9f020])
I20250625 08:04:55.955085 3814 tablet_loader.cc:96] loaded metadata for tablet 499fa735a33d450db3047beb0e018f6f (table default.table_b0a79766b44a4a6882c263b6c2c46ed2 [id=c0c4794ff9864a07a9f6322526e4397a])
I20250625 08:04:55.955257 3814 tablet_loader.cc:96] loaded metadata for tablet 4b5dc84827a0457983d91427688e0c67 (table default.table_23c54834c68a47bd9654b34281b53e7d [id=f060ac216bb54def8fc8a79571936089])
I20250625 08:04:55.955773 3814 tablet_loader.cc:96] loaded metadata for tablet 85074a5f6dfb4c4b9c869a4141175da6 (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206])
I20250625 08:04:55.955941 3814 tablet_loader.cc:96] loaded metadata for tablet 90154412b7b04b6db02f912a53000edb (table default.table_c06b3617aaf24ba4b0ae0b3926081026 [id=874b84d5e2004697a01c0686260a6904])
I20250625 08:04:55.956413 3814 tablet_loader.cc:96] loaded metadata for tablet a0db2bca14734321aa2167a9aade37e0 (table default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d])
I20250625 08:04:55.957399 3814 tablet_loader.cc:96] loaded metadata for tablet db339d8e6f8343d994e0ba656d00acb2 (table default.table_b49cbf7ee55042b39d7b5d3946a35618 [id=1a45dede976343d1be49292f6353b875])
I20250625 08:04:55.957955 3814 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 08:04:55.958791 3814 catalog_manager.cc:1261] Loaded cluster ID: e28fc882d26c4c5dba33e60b796f14e1
I20250625 08:04:55.958842 3814 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 08:04:55.959715 3814 catalog_manager.cc:1506] Loading token signing keys...
I20250625 08:04:55.960350 3814 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Loaded TSK: 0
I20250625 08:04:55.961118 3814 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 08:04:55.961761 3814 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250625 08:04:55.962363 3814 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: 20
I20250625 08:04:56.026474 3397 master_service.cc:948] ReplaceTablet: received request to replace tablet 85074a5f6dfb4c4b9c869a4141175da6 from {username='slave'} at 127.0.0.1:38256
I20250625 08:04:56.027172 3390 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38256:
name: "default.Table_128075ffc1db4314b52b7c28536f6e89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:56.027693 3391 master_service.cc:948] ReplaceTablet: received request to replace tablet a0db2bca14734321aa2167a9aade37e0 from {username='slave'} at 127.0.0.1:38256
I20250625 08:04:56.027751 3395 master_service.cc:948] ReplaceTablet: received request to replace tablet 85074a5f6dfb4c4b9c869a4141175da6 from {username='slave'} at 127.0.0.1:38256
I20250625 08:04:56.027928 3394 master_service.cc:948] ReplaceTablet: received request to replace tablet 85074a5f6dfb4c4b9c869a4141175da6 from {username='slave'} at 127.0.0.1:38256
I20250625 08:04:56.028110 3392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38256:
name: "default.Table_174089cfe43b4dbb9ed1b9c2921f9187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:56.028438 3396 master_service.cc:948] ReplaceTablet: received request to replace tablet 03cc8f661a7f4245a6d1a601e9a5035c from {username='slave'} at 127.0.0.1:38256
I20250625 08:04:56.028491 3388 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38256:
table { table_name: "default.Table_b49cbf7ee55042b39d7b5d3946a35618" } new_table_name: "default.Table_7cb71508c2934e1aa091c71056ae7a01" modify_external_catalogs: true
I20250625 08:04:56.028693 3393 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38256:
name: "default.Table_035cffaa28ec4a838bb218bb37230370"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:56.028823 3389 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38256:
table { table_name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2" } modify_external_catalogs: true
I20250625 08:04:56.028937 3389 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38256:
table { table_name: "default.Table_b0a79766b44a4a6882c263b6c2c46ed2" } modify_external_catalogs: true
W20250625 08:04:56.032262 3379 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc -> Peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Couldn't send request to peer 755093bec4b54a9588598f814c3708e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 08:04:56.033238 3751 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Refusing update from remote peer 5f0b114bd68344b0bd2ee322bd5eccdc: Log matching property violated. Preceding OpId in replica: term: 7 index: 73. Preceding OpId from leader: term: 11 index: 95. (index mismatch)
I20250625 08:04:56.033959 3808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 73, Time since last communication: 0.000s
F20250625 08:04:56.034555 3450 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 23, previous 23; dumping first 4 out of 11 received events: NotificationEvent(eventId=21, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_31023e55a4384ac88f8415871e52d142, message={"server":"","servicePrincipal":"","db":"default","table":"table_31023e55a4384ac88f8415871e52d142","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_31023e55a4384ac88f8415871e52d142\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_31023e55a4384ac88f8415871e52d142\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_31023e55a4384ac88f8415871e52d142\",\"kudu.table_id\":\"7825ea67af2d42b68200097030b08116\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=22, eventTime=1750838669, eventType=ALTER_TABLE, dbName=default, tableName=table_7cb71508c2934e1aa091c71056ae7a01, message={"server":"","servicePrincipal":"","db":"default","table":"table_b49cbf7ee55042b39d7b5d3946a35618","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_b49cbf7ee55042b39d7b5d3946a35618\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_b49cbf7ee55042b39d7b5d3946a35618\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b49cbf7ee55042b39d7b5d3946a35618\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_7cb71508c2934e1aa091c71056ae7a01\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_7cb71508c2934e1aa091c71056ae7a01\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7cb71508c2934e1aa091c71056ae7a01\",\"kudu.table_id\":\"1a45dede976343d1be49292f6353b875\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=DROP_TABLE, dbName=default, tableName=table_d78071777f3646548f7f75cea674ecdc, message={"server":"","servicePrincipal":"","db":"default","table":"table_d78071777f3646548f7f75cea674ecdc","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_d78071777f3646548f7f75cea674ecdc\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838668},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_d78071777f3646548f7f75cea674ecdc\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_d78071777f3646548f7f75cea674ecdc\",\"kudu.table_id\":\"0e43d85f40e545d9b31c5cb3e62f89a8\",\"transient_lastDdlTime\":\"1750838668\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=23, eventTime=1750838669, eventType=CREATE_TABLE, dbName=default, tableName=table_88079d64160a4318bf8ea1807cb3a30d, message={"server":"","servicePrincipal":"","db":"default","table":"table_88079d64160a4318bf8ea1807cb3a30d","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_88079d64160a4318bf8ea1807cb3a30d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750838669},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/warehouse/table_88079d64160a4318bf8ea1807cb3a30d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_88079d64160a4318bf8ea1807cb3a30d\",\"kudu.table_id\":\"7c7e3cbdaf064697a864873ed87a21b6\",\"transient_lastDdlTime\":\"1750838669\",\"comment\":\"\",\"kudu.master_addresses\":\"127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"e28fc882d26c4c5dba33e60b796f14e1\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750838669,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f70be7e3dcd google::LogMessage::Fail() at ??:0
@ 0x7f70be7e7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f70be7e37cc google::LogMessage::Flush() at ??:0
I20250625 08:04:56.043526 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0b114bd68344b0bd2ee322bd5eccdc. Latest consensus state: current_term: 11 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:56.043673 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:56.043771 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:56.043825 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:56.043879 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:56.043929 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:56.043982 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:56.044049 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:56.044740 3808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:56.044860 3808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc [sys.catalog]: This master's current role is: LEADER
@ 0x7f70be7e3a11 google::LogMessage::~LogMessage() at ??:0
I20250625 08:04:56.046293 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } } }
I20250625 08:04:56.046432 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 08:04:56.047025 3396 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Sending DeleteTablet for 3 replicas of tablet 03cc8f661a7f4245a6d1a601e9a5035c
I20250625 08:04:56.047677 3397 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Sending DeleteTablet for 3 replicas of tablet 85074a5f6dfb4c4b9c869a4141175da6
I20250625 08:04:56.048148 3395 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38256:
name: "default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:56.048517 3397 catalog_manager.cc:6652] ReplaceTablet: tablet 85074a5f6dfb4c4b9c869a4141175da6 deleted and replaced by tablet 9cb9da91a94d469aa58575f8f3cee49c
I20250625 08:04:56.048686 3397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38256:
name: "default.Table_31023e55a4384ac88f8415871e52d142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:04:56.049053 3396 catalog_manager.cc:6652] ReplaceTablet: tablet 03cc8f661a7f4245a6d1a601e9a5035c deleted and replaced by tablet 655f74fc2d6e44b89ee733c2002c2cac
I20250625 08:04:56.049252 3396 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38256:
table { table_name: "default.Table_5eebaa7c05694491963fce48525e4681" } new_table_name: "default.Table_69a1c4361ab540a48fd24a5198251c2c" modify_external_catalogs: true
I20250625 08:04:56.049342 3391 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5f0b114bd68344b0bd2ee322bd5eccdc: Sending DeleteTablet for 3 replicas of tablet a0db2bca14734321aa2167a9aade37e0
I20250625 08:04:56.049611 3391 catalog_manager.cc:6652] ReplaceTablet: tablet a0db2bca14734321aa2167a9aade37e0 deleted and replaced by tablet a5eae9e418604ddfae00dff0b8a56ff4
I20250625 08:04:56.049805 3391 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38256:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:56.049924 3391 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38256:
table { table_name: "default.Table_c06b3617aaf24ba4b0ae0b3926081026" } modify_external_catalogs: true
I20250625 08:04:56.050002 3394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38256:
name: "default.Table_bd43623a5f9f40c58ad750234777ba9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f70c377cc91 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250625 08:04:56.056325 3819 mvcc.cc:204] Tried to move back new op lower bound from 7171435298942132224 to 7171435241444790272. Current Snapshot: MvccSnapshot[applied={T|T < 7171435298936807424}]
@ 0x7f70c377bba8 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250625 08:04:56.057819 1685 tablet_service.cc:1515] Processing DeleteTablet for tablet 85074a5f6dfb4c4b9c869a4141175da6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cb9da91a94d469aa58575f8f3cee49c) from {username='slave'} at 127.0.0.1:44980
I20250625 08:04:56.058141 1689 tablet_service.cc:1515] Processing DeleteTablet for tablet a0db2bca14734321aa2167a9aade37e0 with delete_type TABLET_DATA_DELETED (replaced by tablet a5eae9e418604ddfae00dff0b8a56ff4) from {username='slave'} at 127.0.0.1:44980
I20250625 08:04:56.057850 1686 tablet_service.cc:1515] Processing DeleteTablet for tablet 03cc8f661a7f4245a6d1a601e9a5035c with delete_type TABLET_DATA_DELETED (replaced by tablet 655f74fc2d6e44b89ee733c2002c2cac) from {username='slave'} at 127.0.0.1:44980
I20250625 08:04:56.059762 1828 tablet_service.cc:1515] Processing DeleteTablet for tablet 03cc8f661a7f4245a6d1a601e9a5035c with delete_type TABLET_DATA_DELETED (replaced by tablet 655f74fc2d6e44b89ee733c2002c2cac) from {username='slave'} at 127.0.0.1:33634
I20250625 08:04:56.060030 1826 tablet_service.cc:1515] Processing DeleteTablet for tablet 85074a5f6dfb4c4b9c869a4141175da6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cb9da91a94d469aa58575f8f3cee49c) from {username='slave'} at 127.0.0.1:33634
I20250625 08:04:56.060154 1823 tablet_service.cc:1515] Processing DeleteTablet for tablet a0db2bca14734321aa2167a9aade37e0 with delete_type TABLET_DATA_DELETED (replaced by tablet a5eae9e418604ddfae00dff0b8a56ff4) from {username='slave'} at 127.0.0.1:33634
I20250625 08:04:56.060464 1824 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb9da91a94d469aa58575f8f3cee49c (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:56.060767 1824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb9da91a94d469aa58575f8f3cee49c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:56.060729 1686 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb9da91a94d469aa58575f8f3cee49c (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f70c377b30b _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20250625 08:04:56.060967 1686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb9da91a94d469aa58575f8f3cee49c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:56.063409 3836 tablet_replica.cc:331] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:56.063452 1689 tablet_service.cc:1468] Processing CreateTablet for tablet 655f74fc2d6e44b89ee733c2002c2cac (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:56.063551 3836 raft_consensus.cc:2241] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:56.063664 3836 raft_consensus.cc:2270] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:56.063678 1689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 655f74fc2d6e44b89ee733c2002c2cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:56.064088 3836 ts_tablet_manager.cc:1905] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f70c3780a12 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56482787e752 std::function<>::operator()() at ??:0
I20250625 08:04:56.066179 3836 ts_tablet_manager.cc:1918] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:56.066248 3836 log.cc:1199] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/85074a5f6dfb4c4b9c869a4141175da6
I20250625 08:04:56.066329 1823 tablet_service.cc:1468] Processing CreateTablet for tablet 655f74fc2d6e44b89ee733c2002c2cac (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:56.066581 3836 ts_tablet_manager.cc:1939] T 85074a5f6dfb4c4b9c869a4141175da6 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:56.066651 1823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 655f74fc2d6e44b89ee733c2002c2cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:56.067203 3836 tablet_replica.cc:331] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:56.067319 3836 raft_consensus.cc:2241] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:56.067303 3381 catalog_manager.cc:4928] TS eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085): tablet 85074a5f6dfb4c4b9c869a4141175da6 (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]) successfully deleted
I20250625 08:04:56.067409 3836 raft_consensus.cc:2270] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:56.067781 3836 ts_tablet_manager.cc:1905] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:56.069806 3836 ts_tablet_manager.cc:1918] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:56.069861 3836 log.cc:1199] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/a0db2bca14734321aa2167a9aade37e0
I20250625 08:04:56.070111 3836 ts_tablet_manager.cc:1939] T a0db2bca14734321aa2167a9aade37e0 P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:56.070390 3837 tablet_replica.cc:331] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:56.070544 3837 raft_consensus.cc:2241] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:56.070602 1962 tablet_service.cc:1515] Processing DeleteTablet for tablet 85074a5f6dfb4c4b9c869a4141175da6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cb9da91a94d469aa58575f8f3cee49c) from {username='slave'} at 127.0.0.1:37384
I20250625 08:04:56.070660 3837 raft_consensus.cc:2270] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:56.070803 1689 tablet_service.cc:1468] Processing CreateTablet for tablet a5eae9e418604ddfae00dff0b8a56ff4 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:56.070899 1961 tablet_service.cc:1515] Processing DeleteTablet for tablet 03cc8f661a7f4245a6d1a601e9a5035c with delete_type TABLET_DATA_DELETED (replaced by tablet 655f74fc2d6e44b89ee733c2002c2cac) from {username='slave'} at 127.0.0.1:37384
I20250625 08:04:56.071024 1960 tablet_service.cc:1515] Processing DeleteTablet for tablet a0db2bca14734321aa2167a9aade37e0 with delete_type TABLET_DATA_DELETED (replaced by tablet a5eae9e418604ddfae00dff0b8a56ff4) from {username='slave'} at 127.0.0.1:37384
I20250625 08:04:56.071041 1689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5eae9e418604ddfae00dff0b8a56ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:56.071072 3837 ts_tablet_manager.cc:1905] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:56.071336 1959 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb9da91a94d469aa58575f8f3cee49c (DEFAULT_TABLE table=default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:56.071619 1959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb9da91a94d469aa58575f8f3cee49c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:56.073289 3837 ts_tablet_manager.cc:1918] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:56.073364 3837 log.cc:1199] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/03cc8f661a7f4245a6d1a601e9a5035c
I20250625 08:04:56.073554 1823 tablet_service.cc:1468] Processing CreateTablet for tablet a5eae9e418604ddfae00dff0b8a56ff4 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:56.073616 3837 ts_tablet_manager.cc:1939] T 03cc8f661a7f4245a6d1a601e9a5035c P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:56.073786 1823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5eae9e418604ddfae00dff0b8a56ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:56.074272 3837 tablet_replica.cc:331] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:56.074388 3837 raft_consensus.cc:2241] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:56.074600 3837 raft_consensus.cc:2270] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:56.074667 1958 tablet_service.cc:1468] Processing CreateTablet for tablet a5eae9e418604ddfae00dff0b8a56ff4 (DEFAULT_TABLE table=default.table_5eebaa7c05694491963fce48525e4681 [id=8226dc4b416b4b458db9460ca40cd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:56.074956 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5eae9e418604ddfae00dff0b8a56ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:56.075001 3837 ts_tablet_manager.cc:1905] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:56.075384 3381 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 03cc8f661a7f4245a6d1a601e9a5035c (table default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]) successfully deleted
@ 0x7f70bfc350b7 kudu::Thread::SuperviseThread() at ??:0
I20250625 08:04:56.076612 3840 tablet_replica.cc:331] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:56.076776 3840 raft_consensus.cc:2241] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:56.076892 3840 raft_consensus.cc:2270] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:56.076916 3837 ts_tablet_manager.cc:1918] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:56.076972 3837 log.cc:1199] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/85074a5f6dfb4c4b9c869a4141175da6
I20250625 08:04:56.077239 3837 ts_tablet_manager.cc:1939] T 85074a5f6dfb4c4b9c869a4141175da6 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:56.077322 3840 ts_tablet_manager.cc:1905] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:56.077788 3837 tablet_replica.cc:331] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: stopping tablet replica
I20250625 08:04:56.077900 3837 raft_consensus.cc:2241] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:56.077881 3381 catalog_manager.cc:4928] TS de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): tablet 85074a5f6dfb4c4b9c869a4141175da6 (table default.table_d28f89eb30604a7abf867f68131b2c1a [id=b4ad9fa057114ca4878623901a7a5206]) successfully deleted
I20250625 08:04:56.078089 3837 raft_consensus.cc:2270] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f70c04246db start_thread at ??:0
I20250625 08:04:56.078474 3837 ts_tablet_manager.cc:1905] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:56.080440 3837 ts_tablet_manager.cc:1918] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:56.080472 3840 ts_tablet_manager.cc:1918] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:56.080513 3837 log.cc:1199] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-1/wal/wals/a0db2bca14734321aa2167a9aade37e0
I20250625 08:04:56.080539 3840 log.cc:1199] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/85074a5f6dfb4c4b9c869a4141175da6
@ 0x7f70bd99a71f clone at ??:0
I20250625 08:04:56.081180 3840 ts_tablet_manager.cc:1939] T 85074a5f6dfb4c4b9c869a4141175da6 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:56.081343 3839 tablet_bootstrap.cc:492] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:56.081398 1963 tablet_service.cc:1468] Processing CreateTablet for tablet 655f74fc2d6e44b89ee733c2002c2cac (DEFAULT_TABLE table=default.table_88079d64160a4318bf8ea1807cb3a30d [id=7c7e3cbdaf064697a864873ed87a21b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:04:56.081665 1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 655f74fc2d6e44b89ee733c2002c2cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:04:56.081663 3836 tablet_replica.cc:331] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: stopping tablet replica
I20250625 08:04:56.081735 3840 tablet_replica.cc:331] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:56.081795 3836 raft_consensus.cc:2241] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:56.081835 3840 raft_consensus.cc:2241] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Raft consensus shutting down.
I20250625 08:04:56.081902 3836 raft_consensus.cc:2270] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:56.082019 3840 raft_consensus.cc:2270] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:56.082294 3836 ts_tablet_manager.cc:1905] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:56.082351 3840 ts_tablet_manager.cc:1905] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:56.082465 3839 tablet_bootstrap.cc:654] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:56.083212 3838 tablet_bootstrap.cc:492] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:56.084245 3840 ts_tablet_manager.cc:1918] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:56.084322 3840 log.cc:1199] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/03cc8f661a7f4245a6d1a601e9a5035c
I20250625 08:04:56.084391 3838 tablet_bootstrap.cc:654] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:56.084534 3836 ts_tablet_manager.cc:1918] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:56.084575 3840 ts_tablet_manager.cc:1939] T 03cc8f661a7f4245a6d1a601e9a5035c P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:56.084597 3836 log.cc:1199] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-0/wal/wals/03cc8f661a7f4245a6d1a601e9a5035c
I20250625 08:04:56.084745 3837 ts_tablet_manager.cc:1939] T a0db2bca14734321aa2167a9aade37e0 P de62034f80d9463dbd6386fc3e90fdaf: Deleting consensus metadata
I20250625 08:04:56.084826 3836 ts_tablet_manager.cc:1939] T 03cc8f661a7f4245a6d1a601e9a5035c P eebfa8472fe04080ac012e2c91bedaae: Deleting consensus metadata
I20250625 08:04:56.085808 3840 tablet_replica.cc:331] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: stopping tablet replica
I20250625 08:04:56.085959 3840 raft_consensus.cc:2241] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 08:04:56.086086 3840 raft_consensus.cc:2270] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 08:04:56.086146 3839 tablet_bootstrap.cc:492] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:56.086248 3839 ts_tablet_manager.cc:1397] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 08:04:56.086527 3840 ts_tablet_manager.cc:1905] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 08:04:56.086936 3839 raft_consensus.cc:357] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.087050 3839 raft_consensus.cc:383] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:56.087097 3839 raft_consensus.cc:738] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:56.087292 3839 consensus_queue.cc:260] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.087584 3839 ts_tablet_manager.cc:1428] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:56.087746 3839 tablet_bootstrap.cc:492] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:56.088876 3839 tablet_bootstrap.cc:654] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:56.089188 3840 ts_tablet_manager.cc:1918] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 08:04:56.089257 3840 log.cc:1199] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: Deleting WAL directory at /tmp/dist-test-taskYJaGNk/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750838649158245-1101-0/minicluster-data/ts-2/wal/wals/a0db2bca14734321aa2167a9aade37e0
I20250625 08:04:56.089536 3840 ts_tablet_manager.cc:1939] T a0db2bca14734321aa2167a9aade37e0 P 5dae8595b9484b57bc7ccea543f83130: Deleting consensus metadata
I20250625 08:04:56.089855 3845 tablet_bootstrap.cc:492] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:56.090948 3845 tablet_bootstrap.cc:654] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:56.092212 3845 tablet_bootstrap.cc:492] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:56.092291 3838 tablet_bootstrap.cc:492] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:56.092312 3845 ts_tablet_manager.cc:1397] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250625 08:04:56.092391 3838 ts_tablet_manager.cc:1397] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 08:04:56.092833 3845 raft_consensus.cc:357] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.092945 3845 raft_consensus.cc:383] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:56.092978 3845 raft_consensus.cc:738] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:56.092996 3838 raft_consensus.cc:357] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.093128 3838 raft_consensus.cc:383] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:56.093123 3845 consensus_queue.cc:260] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.093169 3838 raft_consensus.cc:738] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:56.093313 3838 consensus_queue.cc:260] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.093427 3845 ts_tablet_manager.cc:1428] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:56.093642 3838 ts_tablet_manager.cc:1428] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 08:04:56.093833 3838 tablet_bootstrap.cc:492] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:56.094951 3838 tablet_bootstrap.cc:654] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:56.095129 3845 tablet_bootstrap.cc:492] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:56.096182 3845 tablet_bootstrap.cc:654] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:56.096398 3839 tablet_bootstrap.cc:492] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:56.096493 3839 ts_tablet_manager.cc:1397] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 08:04:56.097390 3839 raft_consensus.cc:357] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.097512 3839 raft_consensus.cc:383] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:56.097553 3839 raft_consensus.cc:738] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:56.097733 3839 consensus_queue.cc:260] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.097944 3838 tablet_bootstrap.cc:492] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:56.098028 3838 ts_tablet_manager.cc:1397] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250625 08:04:56.098184 3839 ts_tablet_manager.cc:1428] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 08:04:56.098379 3839 tablet_bootstrap.cc:492] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf: Bootstrap starting.
I20250625 08:04:56.098563 3838 raft_consensus.cc:357] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.098830 3838 raft_consensus.cc:383] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:56.098873 3838 raft_consensus.cc:738] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:56.099059 3838 consensus_queue.cc:260] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.099356 3838 ts_tablet_manager.cc:1428] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:56.099529 3839 tablet_bootstrap.cc:654] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:56.099570 3838 tablet_bootstrap.cc:492] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae: Bootstrap starting.
I20250625 08:04:56.102057 3845 tablet_bootstrap.cc:492] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:56.102173 3845 ts_tablet_manager.cc:1397] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250625 08:04:56.102785 3845 raft_consensus.cc:357] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.102906 3845 raft_consensus.cc:383] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:56.102941 3845 raft_consensus.cc:738] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:56.103091 3845 consensus_queue.cc:260] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.103422 3845 ts_tablet_manager.cc:1428] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 08:04:56.103615 3845 tablet_bootstrap.cc:492] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130: Bootstrap starting.
I20250625 08:04:56.104727 3845 tablet_bootstrap.cc:654] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:56.105453 3838 tablet_bootstrap.cc:654] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae: Neither blocks nor log segments found. Creating new log.
I20250625 08:04:56.106907 3839 tablet_bootstrap.cc:492] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf: No bootstrap required, opened a new log
I20250625 08:04:56.107048 3839 ts_tablet_manager.cc:1397] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250625 08:04:56.107867 3839 raft_consensus.cc:357] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.107991 3839 raft_consensus.cc:383] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:56.108031 3839 raft_consensus.cc:738] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Initialized, Role: FOLLOWER
I20250625 08:04:56.108247 3839 consensus_queue.cc:260] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.108577 3845 tablet_bootstrap.cc:492] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130: No bootstrap required, opened a new log
I20250625 08:04:56.108669 3845 ts_tablet_manager.cc:1397] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250625 08:04:56.108709 3839 ts_tablet_manager.cc:1428] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 08:04:56.109242 3845 raft_consensus.cc:357] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.109355 3845 raft_consensus.cc:383] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:56.109388 3845 raft_consensus.cc:738] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Initialized, Role: FOLLOWER
I20250625 08:04:56.109541 3845 consensus_queue.cc:260] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.109624 3838 tablet_bootstrap.cc:492] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae: No bootstrap required, opened a new log
I20250625 08:04:56.110091 3838 ts_tablet_manager.cc:1397] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250625 08:04:56.111127 3845 ts_tablet_manager.cc:1428] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 08:04:56.111742 3838 raft_consensus.cc:357] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.112110 3838 raft_consensus.cc:383] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:04:56.112160 3838 raft_consensus.cc:738] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebfa8472fe04080ac012e2c91bedaae, State: Initialized, Role: FOLLOWER
I20250625 08:04:56.112298 3838 consensus_queue.cc:260] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.113330 3838 ts_tablet_manager.cc:1428] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 08:04:56.114070 3336 raft_consensus.cc:491] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:56.114164 3336 raft_consensus.cc:513] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.114565 3336 leader_election.cc:290] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:56.114928 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5eae9e418604ddfae00dff0b8a56ff4" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:56.114966 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5eae9e418604ddfae00dff0b8a56ff4" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf" is_pre_election: true
I20250625 08:04:56.115025 1711 raft_consensus.cc:2466] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:56.115103 1848 raft_consensus.cc:2466] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 0.
I20250625 08:04:56.115355 1919 leader_election.cc:304] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:56.115495 3336 raft_consensus.cc:2802] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:56.115540 3336 raft_consensus.cc:491] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:56.115584 3336 raft_consensus.cc:3058] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:56.116487 3336 raft_consensus.cc:513] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.116945 3336 leader_election.cc:290] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Requested vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:56.117385 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5eae9e418604ddfae00dff0b8a56ff4" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:56.117421 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5eae9e418604ddfae00dff0b8a56ff4" candidate_uuid: "5dae8595b9484b57bc7ccea543f83130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de62034f80d9463dbd6386fc3e90fdaf"
I20250625 08:04:56.117487 1711 raft_consensus.cc:3058] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:56.117547 1848 raft_consensus.cc:3058] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:56.118258 1711 raft_consensus.cc:2466] T a5eae9e418604ddfae00dff0b8a56ff4 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:56.118636 1848 raft_consensus.cc:2466] T a5eae9e418604ddfae00dff0b8a56ff4 P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dae8595b9484b57bc7ccea543f83130 in term 1.
I20250625 08:04:56.118770 1919 leader_election.cc:304] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:56.119230 3336 raft_consensus.cc:2802] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:56.119433 3336 raft_consensus.cc:695] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [term 1 LEADER]: Becoming Leader. State: Replica: 5dae8595b9484b57bc7ccea543f83130, State: Running, Role: LEADER
I20250625 08:04:56.119616 3336 consensus_queue.cc:237] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.192003 3697 raft_consensus.cc:491] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:56.192152 3697 raft_consensus.cc:513] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.192628 3697 leader_election.cc:290] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:56.193076 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "655f74fc2d6e44b89ee733c2002c2cac" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:56.193181 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "655f74fc2d6e44b89ee733c2002c2cac" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:56.193264 1983 raft_consensus.cc:2466] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:56.193301 1711 raft_consensus.cc:2466] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:56.193598 1784 leader_election.cc:304] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:56.193761 3697 raft_consensus.cc:2802] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:56.193820 3697 raft_consensus.cc:491] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:56.193874 3697 raft_consensus.cc:3058] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:56.194909 3697 raft_consensus.cc:513] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.195273 3697 leader_election.cc:290] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:56.195659 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "655f74fc2d6e44b89ee733c2002c2cac" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:56.195744 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "655f74fc2d6e44b89ee733c2002c2cac" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:56.195796 1983 raft_consensus.cc:3058] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:56.195856 1711 raft_consensus.cc:3058] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:56.196810 1711 raft_consensus.cc:2466] T 655f74fc2d6e44b89ee733c2002c2cac P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:56.196811 1983 raft_consensus.cc:2466] T 655f74fc2d6e44b89ee733c2002c2cac P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:56.197170 1784 leader_election.cc:304] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de62034f80d9463dbd6386fc3e90fdaf, eebfa8472fe04080ac012e2c91bedaae; no voters:
I20250625 08:04:56.197367 3697 raft_consensus.cc:2802] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:56.197582 3697 raft_consensus.cc:695] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:56.197762 3697 consensus_queue.cc:237] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
W20250625 08:04:56.266963 1781 connection.cc:537] server connection from 127.0.0.1:33634 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:56.266963 1644 connection.cc:537] server connection from 127.0.0.1:44980 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:56.267005 1917 connection.cc:537] server connection from 127.0.0.1:37384 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:56.267292 1784 connection.cc:537] client connection to 127.1.19.126:36875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:56.267350 1647 connection.cc:537] client connection to 127.1.19.126:36875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:56.267844 2039 connection.cc:537] client connection to 127.1.19.126:36875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:56.268077 1919 connection.cc:537] client connection to 127.1.19.126:36875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 08:04:56.268286 2039 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.1.19.126:36875)
W20250625 08:04:56.268648 2039 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.19.126:36875)
W20250625 08:04:56.268801 2039 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.1.19.126:36875)
W20250625 08:04:56.268951 2039 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.1.19.126:36875)
I20250625 08:04:56.296305 3697 raft_consensus.cc:491] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:04:56.296460 3697 raft_consensus.cc:513] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.297314 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb9da91a94d469aa58575f8f3cee49c" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130" is_pre_election: true
I20250625 08:04:56.297477 1983 raft_consensus.cc:2466] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:56.297513 3697 leader_election.cc:290] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:56.297793 1784 leader_election.cc:304] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:56.297950 3697 raft_consensus.cc:2802] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:04:56.298004 3697 raft_consensus.cc:491] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:04:56.297978 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb9da91a94d469aa58575f8f3cee49c" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae" is_pre_election: true
I20250625 08:04:56.298063 3697 raft_consensus.cc:3058] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:56.298102 1711 raft_consensus.cc:2466] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 0.
I20250625 08:04:56.299016 3697 raft_consensus.cc:513] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.299371 3697 leader_election.cc:290] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Requested vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), eebfa8472fe04080ac012e2c91bedaae (127.1.19.65:46085)
I20250625 08:04:56.299670 1983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb9da91a94d469aa58575f8f3cee49c" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dae8595b9484b57bc7ccea543f83130"
I20250625 08:04:56.299791 1983 raft_consensus.cc:3058] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:56.299767 1711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb9da91a94d469aa58575f8f3cee49c" candidate_uuid: "de62034f80d9463dbd6386fc3e90fdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebfa8472fe04080ac012e2c91bedaae"
I20250625 08:04:56.299878 1711 raft_consensus.cc:3058] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:04:56.300586 1983 raft_consensus.cc:2466] T 9cb9da91a94d469aa58575f8f3cee49c P 5dae8595b9484b57bc7ccea543f83130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:56.300601 1711 raft_consensus.cc:2466] T 9cb9da91a94d469aa58575f8f3cee49c P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de62034f80d9463dbd6386fc3e90fdaf in term 1.
I20250625 08:04:56.300843 1784 leader_election.cc:304] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf; no voters:
I20250625 08:04:56.301004 3697 raft_consensus.cc:2802] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:04:56.301095 3697 raft_consensus.cc:695] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [term 1 LEADER]: Becoming Leader. State: Replica: de62034f80d9463dbd6386fc3e90fdaf, State: Running, Role: LEADER
I20250625 08:04:56.301241 3697 consensus_queue.cc:237] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } }
I20250625 08:04:56.558499 3857 consensus_queue.cc:1035] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:56.563798 3336 consensus_queue.cc:1035] T a5eae9e418604ddfae00dff0b8a56ff4 P 5dae8595b9484b57bc7ccea543f83130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:56.569308 3876 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:56.569449 3876 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:56.569856 3876 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:56.571103 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:04:56.572018 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:56.572088 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:56.572297 3876 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:04:56.580220 3697 consensus_queue.cc:1035] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:56.584664 3697 consensus_queue.cc:1035] T 655f74fc2d6e44b89ee733c2002c2cac P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:56.790067 3882 consensus_queue.cc:1035] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:56.794593 3882 consensus_queue.cc:1035] T 9cb9da91a94d469aa58575f8f3cee49c P de62034f80d9463dbd6386fc3e90fdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:04:57.082864 3888 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:57.083007 3888 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:57.083513 3888 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:57.084767 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:04:57.084854 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:57.084952 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:57.085105 3888 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:04:57.620182 1101 master-stress-test.cc:577] Killing test thread 1/25
I20250625 08:04:58.209558 3904 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:58.209689 3904 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:58.210057 3904 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:58.211305 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:04:58.211385 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:58.211452 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:58.211589 3904 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:04:58.862684 3915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:04:58.862823 3915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:04:58.863284 3915 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:04:58.864709 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:04:58.864799 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:04:58.864864 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:04:58.865000 3915 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:05:00.363153 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:00.363298 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:00.648799 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 26: this message will repeat every 5th retry.
I20250625 08:05:00.740731 3946 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:05:00.740866 3946 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:05:00.741293 3946 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:05:00.741694 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20250625 08:05:00.742466 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:05:00.742599 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:05:00.742688 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:05:00.742849 3946 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:05:00.751866 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 26: this message will repeat every 5th retry.
I20250625 08:05:01.296304 3955 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:05:01.296449 3955 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:05:01.296904 3955 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:05:01.298077 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:05:01.298273 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:05:01.298411 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:05:01.298592 3955 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:05:02.901888 3481 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:02.902051 3481 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:05:02.902662 3481 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:05:02.903556 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:05:02.903712 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:05:02.903759 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:02.903890 3481 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:05:03.609212 3921 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:03.609385 3921 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:05:03.609853 3921 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:05:03.610606 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:05:03.610731 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:05:03.610806 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:03.610991 3921 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:05:05.364434 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:05.364573 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:05.553646 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 31: this message will repeat every 5th retry.
W20250625 08:05:05.795433 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 31: this message will repeat every 5th retry.
I20250625 08:05:05.840227 4005 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:05:05.840378 4005 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:05:05.840860 4005 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:05:05.841274 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250625 08:05:05.842003 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:05:05.842078 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:05:05.842139 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:05:05.842267 4005 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:05:08.599776 4021 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:08.599933 4021 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:05:08.600359 4021 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:05:08.601128 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:05:08.601269 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:05:08.601327 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:08.601485 4021 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:05:10.365721 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:10.365895 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:10.562414 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 36: this message will repeat every 5th retry.
W20250625 08:05:10.776813 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 36: this message will repeat every 5th retry.
I20250625 08:05:12.059921 4081 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:05:12.060052 4081 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:05:12.060506 4081 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:05:12.061054 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:05:12.061709 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:05:12.061802 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:05:12.061882 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:05:12.062011 4081 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:05:15.137296 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 41: this message will repeat every 5th retry.
I20250625 08:05:15.366783 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:15.366904 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:15.584992 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 41: this message will repeat every 5th retry.
I20250625 08:05:16.882982 3921 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:16.883132 3921 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:05:16.883533 3921 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:05:16.884196 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:05:16.884346 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:05:16.884405 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:16.884545 3921 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:05:20.173071 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 46: this message will repeat every 5th retry.
I20250625 08:05:20.368237 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:20.368366 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:20.434705 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 46: this message will repeat every 5th retry.
I20250625 08:05:20.593056 4180 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:05:20.593204 4180 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:05:20.593645 4180 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:05:20.594110 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:05:20.594833 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:05:20.594946 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:05:20.595050 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:05:20.595198 4180 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:05:20.625602 3921 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:20.625763 3921 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:05:20.626300 3921 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:05:20.627075 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:05:20.627200 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:05:20.627256 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:20.627409 3921 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:05:25.369429 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:25.369560 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:25.378226 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 51: this message will repeat every 5th retry.
W20250625 08:05:25.467342 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 51: this message will repeat every 5th retry.
W20250625 08:05:27.392964 1757 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (165 consecutive failures): Network error: Failed to ping master at 127.1.19.124:37745: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:05:27.788141 1895 heartbeater.cc:646] Failed to heartbeat to 127.1.19.126:36875 (70 consecutive failures): Network error: Failed to ping master at 127.1.19.126:36875: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:05:27.866813 4021 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:27.866926 4021 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:05:27.867282 4021 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:05:27.867877 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:05:27.867998 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:05:27.868052 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:27.868207 4021 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:05:28.096128 2029 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (89 consecutive failures): Network error: Failed to ping master at 127.1.19.124:37745: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:05:29.620702 4020 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:29.620867 4020 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:05:29.621300 4020 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:05:29.622058 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:05:29.622185 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:05:29.622241 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:29.622401 4020 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:05:30.370606 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:30.370733 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:30.386791 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 56: this message will repeat every 5th retry.
W20250625 08:05:30.441658 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 56: this message will repeat every 5th retry.
I20250625 08:05:34.424324 3921 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:34.424465 3921 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:05:34.424937 3921 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:05:34.425784 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:05:34.425889 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:05:34.425925 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:34.426077 3921 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:05:35.104827 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 61: this message will repeat every 5th retry.
I20250625 08:05:35.371789 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:35.371912 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:05:35.424932 4340 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:05:35.425068 4340 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:05:35.425483 4340 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:05:35.426005 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:05:35.426714 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:05:35.426800 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:05:35.426888 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:05:35.427035 4340 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:05:35.540558 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 61: this message will repeat every 5th retry.
I20250625 08:05:39.911823 4020 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:39.911938 4020 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:05:39.912497 4020 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:05:39.913060 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:05:39.913184 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:05:39.913232 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:39.913376 4020 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:05:40.171314 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 66: this message will repeat every 5th retry.
W20250625 08:05:40.227891 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 66: this message will repeat every 5th retry.
I20250625 08:05:40.372968 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:40.373107 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:05:43.472848 4021 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:43.472997 4021 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:05:43.473755 4021 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:05:43.474522 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:05:43.474651 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:05:43.474707 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:43.474841 4021 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:05:44.996737 4446 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:05:44.996874 4446 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:05:44.997323 4446 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:05:44.997872 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:05:44.998508 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:05:44.998636 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:05:44.998713 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:05:44.998879 4446 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:05:45.137099 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 71: this message will repeat every 5th retry.
I20250625 08:05:45.374014 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:45.374132 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:45.613550 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 71: this message will repeat every 5th retry.
I20250625 08:05:47.657564 4020 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:47.657721 4020 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:05:47.658131 4020 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:05:47.659101 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:05:47.659215 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:05:47.659274 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:47.659452 4020 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:05:50.249177 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 76: this message will repeat every 5th retry.
I20250625 08:05:50.375211 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:50.375325 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:50.521076 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 76: this message will repeat every 5th retry.
I20250625 08:05:54.947979 4562 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:05:54.948136 4562 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:05:54.948623 4562 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:05:54.949648 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:05:54.949820 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:05:54.949884 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:05:54.950071 4497 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:05:55.147261 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 81: this message will repeat every 5th retry.
I20250625 08:05:55.376328 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:05:55.376449 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:05:55.419375 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 81: this message will repeat every 5th retry.
W20250625 08:06:00.009052 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 86: this message will repeat every 5th retry.
I20250625 08:06:00.377658 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:00.377794 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:06:00.585743 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 86: this message will repeat every 5th retry.
I20250625 08:06:01.520864 4562 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:01.521003 4562 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:06:01.521430 4562 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:06:01.522135 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:06:01.522236 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:06:01.522275 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:01.522394 4562 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:06:02.455832 4650 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:06:02.455968 4650 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:06:02.456553 4650 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:06:02.457142 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:06:02.457881 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:06:02.457971 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:06:02.458051 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:06:02.458194 4650 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:06:02.972143 4562 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:02.972299 4562 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:06:02.972746 4562 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:06:02.973565 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:06:02.973690 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:06:02.973734 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:02.973891 4562 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:06:04.995491 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 91: this message will repeat every 5th retry.
I20250625 08:06:05.378798 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:05.378904 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:06:05.398098 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 91: this message will repeat every 5th retry.
I20250625 08:06:10.379865 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:10.380013 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:06:10.523816 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 96: this message will repeat every 5th retry.
W20250625 08:06:10.702821 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 96: this message will repeat every 5th retry.
I20250625 08:06:13.654561 4724 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:13.654721 4724 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:06:13.655259 4724 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:06:13.655992 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:06:13.656132 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:06:13.656185 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:13.656345 4724 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:06:15.381249 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:15.381397 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:06:15.440734 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 101: this message will repeat every 5th retry.
W20250625 08:06:15.502465 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 101: this message will repeat every 5th retry.
I20250625 08:06:16.732961 4817 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:06:16.733106 4817 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:06:16.733516 4817 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:06:16.734081 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:06:16.734766 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:06:16.734860 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:06:16.734935 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:06:16.735069 4817 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:06:18.003733 4803 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:18.003875 4803 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:06:18.004314 4803 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:06:18.005120 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:06:18.005231 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:06:18.005268 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:18.005425 4803 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:06:20.131444 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 106: this message will repeat every 5th retry.
W20250625 08:06:20.356122 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 106: this message will repeat every 5th retry.
I20250625 08:06:20.382386 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:20.382542 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:06:22.170276 4855 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:22.170416 4855 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:06:22.170886 4855 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:06:22.171608 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:06:22.171716 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:06:22.171752 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:22.171878 4855 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:06:24.925498 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 111: this message will repeat every 5th retry.
W20250625 08:06:25.156471 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 111: this message will repeat every 5th retry.
I20250625 08:06:25.383591 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:25.383708 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:06:27.443001 1758 heartbeater.cc:646] Failed to heartbeat to 127.1.19.126:36875 (383 consecutive failures): Network error: Failed to ping master at 127.1.19.126:36875: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:06:27.910133 1894 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (209 consecutive failures): Network error: Failed to ping master at 127.1.19.124:37745: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:06:28.217861 2029 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (209 consecutive failures): Network error: Failed to ping master at 127.1.19.124:37745: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:06:29.753865 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 116: this message will repeat every 5th retry.
I20250625 08:06:30.384763 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:30.384865 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:06:30.386482 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 116: this message will repeat every 5th retry.
I20250625 08:06:33.905473 5009 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:06:33.905643 5009 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:06:33.906199 5009 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:06:33.906805 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:06:33.907416 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:06:33.907506 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:06:33.907583 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:06:33.907718 5009 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:06:34.174877 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 121: this message will repeat every 5th retry.
I20250625 08:06:35.385967 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:35.386085 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:06:35.539580 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 121: this message will repeat every 5th retry.
I20250625 08:06:36.014251 4855 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:36.014417 4855 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:06:36.015024 4855 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:06:36.016285 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:06:36.016438 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:06:36.016501 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:36.016651 5037 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:06:36.265300 4855 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:36.265455 4855 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:06:36.265894 4855 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:06:36.266693 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:06:36.266821 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:06:36.266875 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:36.267022 4855 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:06:39.392163 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 126: this message will repeat every 5th retry.
I20250625 08:06:40.390400 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:40.390540 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:06:40.491958 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 126: this message will repeat every 5th retry.
I20250625 08:06:40.828550 5105 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:06:40.828728 5105 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:06:40.829174 5105 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:06:40.829617 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:06:40.830333 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:06:40.830411 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:06:40.830518 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:06:40.830648 5105 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:06:43.486899 5079 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:43.487071 5079 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:06:43.487521 5079 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:06:43.488284 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:06:43.488420 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:06:43.488468 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:43.488652 5079 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:06:44.639014 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 131: this message will repeat every 5th retry.
W20250625 08:06:45.377811 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 131: this message will repeat every 5th retry.
I20250625 08:06:45.391588 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:45.391713 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:06:47.312232 5179 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:06:47.312410 5179 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:06:47.312907 5179 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:06:47.313833 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:06:47.313833 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:06:47.314786 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:06:47.314901 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:06:47.314986 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:06:47.315146 5179 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:06:47.881796 5092 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:47.881982 5092 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:06:47.882493 5092 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:06:47.883374 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:06:47.883523 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:06:47.883584 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:47.883737 5092 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:06:49.767361 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 136: this message will repeat every 5th retry.
I20250625 08:06:50.392779 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:50.392899 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:06:50.718361 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 136: this message will repeat every 5th retry.
I20250625 08:06:51.461004 5093 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:51.461145 5093 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:06:51.461697 5093 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:06:51.462414 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:06:51.462571 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:06:51.462604 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:51.462774 5093 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:06:53.614331 5092 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:53.614496 5092 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:06:53.614881 5092 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:06:53.615623 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:06:53.615743 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:06:53.615794 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:53.615937 5092 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:06:55.006681 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 141: this message will repeat every 5th retry.
I20250625 08:06:55.394059 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:06:55.394182 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:06:55.424448 3730 maintenance_manager.cc:419] P 0404561252194ce29bf2924fd5e67e22: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033357
I20250625 08:06:55.472563 3729 maintenance_manager.cc:643] P 0404561252194ce29bf2924fd5e67e22: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.048s user 0.019s sys 0.000s Metrics: {"bytes_written":23709,"cfile_init":1,"dirs.queue_time_us":300,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":980,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":531,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":56,"thread_start_us":206,"threads_started":2,"wal-append.queue_time_us":173}
I20250625 08:06:55.473511 3730 maintenance_manager.cc:419] P 0404561252194ce29bf2924fd5e67e22: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3217 bytes on disk
I20250625 08:06:55.474087 3729 maintenance_manager.cc:643] P 0404561252194ce29bf2924fd5e67e22: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20250625 08:06:55.580876 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 141: this message will repeat every 5th retry.
I20250625 08:06:57.481307 5298 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:06:57.481492 5298 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:06:57.481987 5298 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:06:57.482950 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:06:57.483070 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:06:57.483130 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:06:57.483379 5298 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:00.080371 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 146: this message will repeat every 5th retry.
I20250625 08:07:00.395347 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:00.395473 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:07:00.479990 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 146: this message will repeat every 5th retry.
I20250625 08:07:00.877242 5298 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:00.877398 5298 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:00.877892 5298 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:00.878753 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:00.878903 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:00.878962 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:00.879128 5298 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:07:02.255219 5281 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:02.255378 5281 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:07:02.255838 5281 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:07:02.256688 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:07:02.256811 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:07:02.256865 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:02.257005 5281 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:05.159982 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 151: this message will repeat every 5th retry.
I20250625 08:07:05.382251 5404 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:07:05.382378 5404 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:07:05.382934 5404 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:07:05.383641 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:07:05.384302 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:07:05.384377 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:07:05.384438 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:07:05.384563 5404 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:07:05.396471 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:05.396603 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:07:05.565491 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 151: this message will repeat every 5th retry.
I20250625 08:07:07.040688 5281 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:07.040834 5281 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:07.041396 5281 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:07.042131 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:07.042289 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:07.042344 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:07.042501 5281 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:07:09.169624 5281 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:09.169787 5281 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:07:09.170194 5281 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:07:09.171001 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:07:09.171124 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:07:09.171172 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:09.171321 5281 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:09.781718 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 156: this message will repeat every 5th retry.
W20250625 08:07:10.340646 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 156: this message will repeat every 5th retry.
I20250625 08:07:10.397727 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:10.397850 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:07:13.782485 5280 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:13.782652 5280 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:13.783100 5280 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:13.783901 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:13.784010 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:13.784045 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:13.784195 5280 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:07:14.250367 5539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:07:14.250555 5539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:07:14.251022 5539 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:07:14.251562 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:07:14.252202 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:07:14.252296 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:07:14.252368 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:07:14.252496 5539 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:07:14.472738 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 161: this message will repeat every 5th retry.
W20250625 08:07:15.288846 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 161: this message will repeat every 5th retry.
I20250625 08:07:15.398721 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:15.398844 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:07:16.769903 5281 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:16.770053 5281 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:07:16.770550 5281 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:07:16.771293 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:07:16.771427 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:07:16.771485 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:16.771615 5281 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:07:19.126297 5280 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:19.126474 5280 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:19.126958 5280 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:19.127775 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:19.127890 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:19.127946 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:19.128105 5280 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:19.207878 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 166: this message will repeat every 5th retry.
W20250625 08:07:20.309731 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 166: this message will repeat every 5th retry.
I20250625 08:07:20.399834 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:20.399958 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:07:21.325001 5280 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:21.325142 5280 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:21.325629 5280 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:21.326401 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:21.326555 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:21.326611 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:21.326789 5280 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:23.909978 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 171: this message will repeat every 5th retry.
W20250625 08:07:25.282343 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 171: this message will repeat every 5th retry.
I20250625 08:07:25.357941 5698 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:25.358125 5698 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:25.358704 5698 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:25.359433 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:25.359558 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:25.359598 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:25.359727 5698 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:07:25.400960 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:25.401073 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:07:26.107193 5716 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:07:26.107331 5716 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:07:26.107722 5716 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:07:26.108283 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:07:26.108924 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:07:26.109026 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:07:26.109104 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:07:26.109251 5716 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:07:26.584507 5280 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:26.584626 5280 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:07:26.584996 5280 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:07:26.585685 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:07:26.585827 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:07:26.585881 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:26.586043 5280 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:27.807250 1757 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (657 consecutive failures): Network error: Failed to ping master at 127.1.19.124:37745: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:07:28.035446 1895 heartbeater.cc:646] Failed to heartbeat to 127.1.19.126:36875 (310 consecutive failures): Network error: Failed to ping master at 127.1.19.126:36875: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:07:28.342754 2029 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (329 consecutive failures): Network error: Failed to ping master at 127.1.19.124:37745: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:07:28.495427 5755 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:07:28.495560 5755 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:07:28.496016 5755 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:07:28.497277 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:07:28.497375 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:07:28.497463 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:07:28.497608 5755 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:07:29.221951 5768 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:29.222111 5768 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:29.222553 5768 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:29.223114 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:29.223220 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:29.223273 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:29.223397 5768 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:29.411725 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 176: this message will repeat every 5th retry.
I20250625 08:07:30.402124 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:30.402257 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:07:30.618391 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 176: this message will repeat every 5th retry.
I20250625 08:07:33.088150 5280 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:33.088289 5280 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:33.088681 5280 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:33.089331 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:33.089480 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:33.089535 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:33.089689 5280 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:34.698853 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 181: this message will repeat every 5th retry.
W20250625 08:07:35.195747 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 181: this message will repeat every 5th retry.
I20250625 08:07:35.403409 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:35.403528 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:07:39.671952 5767 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:39.672103 5767 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:39.672498 5767 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:39.673346 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:39.673485 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:39.673540 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:39.673700 5767 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:39.831404 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 186: this message will repeat every 5th retry.
W20250625 08:07:40.298341 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 186: this message will repeat every 5th retry.
I20250625 08:07:40.404784 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:40.404893 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:07:42.486487 5965 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:07:42.486616 5965 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:07:42.487080 5965 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:07:42.487564 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250625 08:07:42.488184 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:07:42.488270 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:07:42.488348 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:07:42.488477 5965 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:07:44.619309 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 191: this message will repeat every 5th retry.
I20250625 08:07:45.406034 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:45.406148 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:07:45.787065 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 191: this message will repeat every 5th retry.
I20250625 08:07:46.361496 5768 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:46.361663 5768 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:07:46.362192 5768 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:07:46.363019 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:07:46.363144 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:07:46.363196 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:46.363353 5768 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:07:47.638541 6042 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:07:47.638692 6042 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:07:47.639168 6042 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:07:47.639703 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:07:47.640376 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:07:47.640475 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:07:47.640537 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:07:47.640662 6042 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:07:49.940018 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 196: this message will repeat every 5th retry.
I20250625 08:07:50.407079 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:50.407188 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:07:50.619535 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 196: this message will repeat every 5th retry.
I20250625 08:07:51.638856 6088 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:51.639003 6088 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:07:51.639421 6088 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:07:51.640177 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:07:51.640301 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:07:51.640353 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:51.640509 6088 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:07:54.980203 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 201: this message will repeat every 5th retry.
I20250625 08:07:55.410976 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:07:55.411093 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:07:55.563454 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 201: this message will repeat every 5th retry.
I20250625 08:07:57.473212 6193 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:07:57.473364 6193 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:07:57.473826 6193 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:07:57.474402 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:07:57.475152 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:07:57.475236 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:07:57.475317 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:07:57.475453 6193 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:07:59.366670 5767 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:07:59.366819 5767 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:07:59.367259 5767 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:07:59.368115 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:07:59.368222 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:07:59.368283 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:07:59.368456 5767 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:08:00.059674 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 206: this message will repeat every 5th retry.
I20250625 08:08:00.412011 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:00.412128 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:00.675884 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 206: this message will repeat every 5th retry.
I20250625 08:08:00.861230 5767 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:00.861375 5767 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:08:00.861788 5767 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:08:00.862591 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:08:00.862713 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:08:00.862766 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:00.862926 5767 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:08:02.524636 6267 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:08:02.524772 6267 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:08:02.525197 6267 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:08:02.525745 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:08:02.526432 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:08:02.526553 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:08:02.526614 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:08:02.526732 6267 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:08:05.084776 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 211: this message will repeat every 5th retry.
I20250625 08:08:05.413738 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:05.413867 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:05.810977 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 211: this message will repeat every 5th retry.
I20250625 08:08:09.342988 5767 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:09.343225 5767 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:08:09.343765 5767 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:08:09.344368 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:08:09.344488 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:08:09.344540 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:09.344678 5767 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:08:10.114578 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 216: this message will repeat every 5th retry.
I20250625 08:08:10.414899 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:10.415010 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:10.916935 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 216: this message will repeat every 5th retry.
I20250625 08:08:14.382525 6139 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:14.382651 6139 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:08:14.383018 6139 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:08:14.383733 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:08:14.383839 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:08:14.383891 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:14.384052 6139 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:08:15.242523 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 221: this message will repeat every 5th retry.
I20250625 08:08:15.415989 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:15.416096 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:16.021924 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 221: this message will repeat every 5th retry.
I20250625 08:08:18.248788 5767 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:18.248929 5767 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:08:18.249343 5767 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:08:18.250068 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:08:18.250192 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:08:18.250245 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:18.250415 5767 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:08:18.871919 6500 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:08:18.872088 6500 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:08:18.872578 6500 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:08:18.873152 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:08:18.874033 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:08:18.874125 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:08:18.874217 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:08:18.874342 6500 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:08:18.914537 6139 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:18.914702 6139 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:08:18.915175 6139 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:08:18.915871 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:08:18.916009 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:08:18.916065 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:18.916230 6139 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:08:19.890239 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 226: this message will repeat every 5th retry.
I20250625 08:08:20.417119 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:20.417263 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:21.116214 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 226: this message will repeat every 5th retry.
I20250625 08:08:23.646139 6139 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:23.646322 6139 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:08:23.646857 6139 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:08:23.647559 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:08:23.647670 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:08:23.647720 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:23.647889 6139 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:08:24.551453 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 231: this message will repeat every 5th retry.
I20250625 08:08:25.418352 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:25.418493 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:25.664834 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 231: this message will repeat every 5th retry.
W20250625 08:08:28.007221 1758 heartbeater.cc:646] Failed to heartbeat to 127.1.19.126:36875 (874 consecutive failures): Network error: Failed to ping master at 127.1.19.126:36875: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:08:28.156369 1894 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (449 consecutive failures): Network error: Failed to ping master at 127.1.19.124:37745: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:08:28.464125 2029 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (449 consecutive failures): Network error: Failed to ping master at 127.1.19.124:37745: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:08:28.989715 6509 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:28.989871 6509 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:08:28.990379 6509 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:08:28.991112 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:08:28.991223 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:08:28.991258 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:28.991394 6509 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:08:29.496800 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 236: this message will repeat every 5th retry.
I20250625 08:08:30.419456 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:30.419572 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:08:30.485746 6669 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:08:30.485880 6669 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:08:30.486366 6669 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:08:30.486910 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:08:30.487602 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:08:30.487694 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:08:30.487790 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:08:30.487920 6669 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:08:30.576321 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 236: this message will repeat every 5th retry.
I20250625 08:08:31.738503 6139 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:31.738708 6139 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:08:31.739192 6139 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:08:31.739923 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:08:31.740144 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:08:31.740196 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:31.740329 6139 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:08:34.446758 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 241: this message will repeat every 5th retry.
I20250625 08:08:34.555130 6139 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:34.555286 6139 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:08:34.555760 6139 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:08:34.556362 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:08:34.556573 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:08:34.556654 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:34.556780 6139 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:08:35.420603 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:35.420727 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:35.722277 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 241: this message will repeat every 5th retry.
I20250625 08:08:35.764233 6509 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:35.764447 6509 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:08:35.765049 6509 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:08:35.765771 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:08:35.765901 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:08:35.765954 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:35.766111 6509 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:08:38.953595 6787 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:08:38.953734 6787 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:08:38.954210 6787 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:08:38.954788 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:08:38.955416 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:08:38.955506 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:08:38.955579 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:08:38.955713 6787 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:08:39.548029 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 246: this message will repeat every 5th retry.
I20250625 08:08:40.421780 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:40.421903 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:40.752913 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 246: this message will repeat every 5th retry.
W20250625 08:08:44.296830 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 251: this message will repeat every 5th retry.
I20250625 08:08:44.370885 6867 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:08:44.371022 6867 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:08:44.371517 6867 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:08:44.372138 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:08:44.372959 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:08:44.373066 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:08:44.373157 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:08:44.373329 6867 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:08:45.423125 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:45.423240 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:45.829720 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 251: this message will repeat every 5th retry.
I20250625 08:08:46.416164 6897 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:46.416316 6897 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:08:46.416836 6897 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:08:46.417707 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:08:46.417850 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:08:46.417907 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:46.418041 6897 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:08:49.595620 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 256: this message will repeat every 5th retry.
I20250625 08:08:50.424407 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:50.424559 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:50.980535 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 256: this message will repeat every 5th retry.
I20250625 08:08:51.984795 6978 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:08:51.984916 6978 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:08:51.985327 6978 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:08:51.985806 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:08:51.986527 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:08:51.986691 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
I20250625 08:08:51.986749 3728 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:08:51.986876 6978 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:08:54.989391 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 261: this message will repeat every 5th retry.
I20250625 08:08:55.058518 6772 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:08:55.058677 6772 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:08:55.059129 6772 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:08:55.059949 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:08:55.060079 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:08:55.060130 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:08:55.060279 6772 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:08:55.425617 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:08:55.425747 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:08:56.187888 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 261: this message will repeat every 5th retry.
W20250625 08:08:59.905400 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 266: this message will repeat every 5th retry.
I20250625 08:09:00.426934 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:00.427063 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:09:00.807443 6139 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:00.807601 6139 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:09:00.808049 6139 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:09:00.808727 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:09:00.808845 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:09:00.808890 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:00.809028 6139 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:09:00.901737 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 266: this message will repeat every 5th retry.
W20250625 08:09:04.590745 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 271: this message will repeat every 5th retry.
I20250625 08:09:05.428149 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:05.428272 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:09:05.846395 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 271: this message will repeat every 5th retry.
I20250625 08:09:07.536279 6897 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:07.536423 6897 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:09:07.536931 6897 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:09:07.537590 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:09:07.537709 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:09:07.537765 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:07.538043 6897 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:09:09.568135 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 276: this message will repeat every 5th retry.
I20250625 08:09:10.429383 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:10.429497 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:09:10.773727 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 276: this message will repeat every 5th retry.
I20250625 08:09:10.950335 6139 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:10.950534 6139 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:09:10.950940 6139 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:09:10.951824 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:09:10.951928 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:09:10.951982 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:10.952158 6139 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:09:11.159932 7241 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:09:11.160095 7241 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:09:11.160759 7241 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:09:11.161410 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:09:11.162078 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:11.162186 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:09:11.162285 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:09:11.162461 7241 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:09:14.092284 6139 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:14.092465 6139 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:09:14.092970 6139 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:09:14.093874 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:09:14.093992 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:09:14.094029 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:14.094183 6139 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:09:14.562999 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 281: this message will repeat every 5th retry.
W20250625 08:09:15.306917 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 281: this message will repeat every 5th retry.
I20250625 08:09:15.430495 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:15.430617 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:09:16.505844 6772 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:16.506006 6772 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:09:16.506543 6772 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:09:16.507364 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:09:16.507493 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:09:16.507551 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:16.507735 6772 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:09:17.384446 6139 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:17.384608 6139 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:09:17.385013 6139 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:09:17.385830 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:09:17.385970 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:09:17.386018 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:17.386196 6139 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:09:19.572073 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 286: this message will repeat every 5th retry.
I20250625 08:09:20.431577 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:20.431712 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:09:20.551112 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 286: this message will repeat every 5th retry.
I20250625 08:09:21.028978 7380 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:09:21.029122 7380 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:09:21.029610 7380 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:09:21.030160 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:09:21.031114 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:21.031224 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:09:21.031288 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:09:21.031445 7380 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:09:24.479187 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 291: this message will repeat every 5th retry.
I20250625 08:09:25.432715 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:25.432834 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:09:25.945429 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 291: this message will repeat every 5th retry.
W20250625 08:09:28.278849 1895 heartbeater.cc:646] Failed to heartbeat to 127.1.19.126:36875 (550 consecutive failures): Network error: Failed to ping master at 127.1.19.126:36875: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:28.362181 1758 heartbeater.cc:646] Failed to heartbeat to 127.1.19.126:36875 (1115 consecutive failures): Network error: Failed to ping master at 127.1.19.126:36875: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:28.588306 2029 heartbeater.cc:646] Failed to heartbeat to 127.1.19.124:37745 (569 consecutive failures): Network error: Failed to ping master at 127.1.19.124:37745: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
W20250625 08:09:29.404157 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 296: this message will repeat every 5th retry.
I20250625 08:09:30.292078 7349 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:30.292234 7349 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:09:30.292769 7349 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:09:30.293527 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:09:30.293689 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:09:30.293735 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:30.293918 7349 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:09:30.433990 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:30.434109 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:09:30.697022 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 296: this message will repeat every 5th retry.
W20250625 08:09:34.495802 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 301: this message will repeat every 5th retry.
I20250625 08:09:34.696764 6772 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:34.696893 6772 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:09:34.697307 6772 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:09:34.697990 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:09:34.698092 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:09:34.698127 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:34.698266 6772 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:09:35.355476 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 301: this message will repeat every 5th retry.
I20250625 08:09:35.435278 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:35.435413 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:09:36.342947 7349 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:36.343135 7349 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:09:36.343593 7349 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:09:36.344365 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:09:36.344484 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:09:36.344540 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:36.344709 7349 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:09:37.856673 7610 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:09:37.856817 7610 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:09:37.857461 7610 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:09:37.858037 3728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250625 08:09:37.858790 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:37.858960 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:09:37.859037 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:09:37.859180 7610 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:09:38.635215 7626 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:09:38.635349 7626 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:09:38.635800 7626 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:09:38.636866 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:38.636956 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:09:38.637049 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:09:38.637181 7626 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:09:38.943706 6897 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:38.943841 6897 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:09:38.944243 6897 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:09:38.944862 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:09:38.945019 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:09:38.945078 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:38.945211 6897 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:09:39.694828 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 306: this message will repeat every 5th retry.
W20250625 08:09:40.419941 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 306: this message will repeat every 5th retry.
I20250625 08:09:40.436467 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:40.436589 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:09:44.414126 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 311: this message will repeat every 5th retry.
W20250625 08:09:45.347339 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 311: this message will repeat every 5th retry.
I20250625 08:09:45.437724 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:45.437880 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 08:09:48.358814 7765 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:48.358968 7765 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:09:48.359508 7765 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:09:48.360284 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:09:48.360385 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:09:48.360421 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:48.360559 7765 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 08:09:49.594696 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 316: this message will repeat every 5th retry.
I20250625 08:09:49.869390 7798 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:09:49.869535 7798 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:09:49.869993 7798 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:09:49.870599 3726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250625 08:09:49.871220 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:49.871353 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:09:49.871436 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:09:49.871585 7798 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250625 08:09:50.335052 7807 raft_consensus.cc:491] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:50.335244 7807 raft_consensus.cc:513] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } } peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } }
I20250625 08:09:50.335704 7807 leader_election.cc:290] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005), de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071)
W20250625 08:09:50.336455 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
W20250625 08:09:50.336571 1647 leader_election.cc:343] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 579f822528014659a13b9dc93111624f
I20250625 08:09:50.336613 1647 leader_election.cc:304] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:50.336751 7807 raft_consensus.cc:2747] T 579f822528014659a13b9dc93111624f P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:09:50.439110 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:50.439247 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:09:50.721877 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 316: this message will repeat every 5th retry.
I20250625 08:09:51.677357 7817 raft_consensus.cc:491] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dae8595b9484b57bc7ccea543f83130)
I20250625 08:09:51.677505 7817 raft_consensus.cc:513] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebfa8472fe04080ac012e2c91bedaae" member_type: VOTER last_known_addr { host: "127.1.19.65" port: 46085 } } peers { permanent_uuid: "de62034f80d9463dbd6386fc3e90fdaf" member_type: VOTER last_known_addr { host: "127.1.19.66" port: 46071 } } peers { permanent_uuid: "5dae8595b9484b57bc7ccea543f83130" member_type: VOTER last_known_addr { host: "127.1.19.67" port: 37005 } }
I20250625 08:09:51.677865 7817 leader_election.cc:290] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071), 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005)
W20250625 08:09:51.678529 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
W20250625 08:09:51.678663 1647 leader_election.cc:343] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Not found: Tablet not found: 18ea19fdd2814cc08c6315b3463bd3a9
I20250625 08:09:51.678711 1647 leader_election.cc:304] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eebfa8472fe04080ac012e2c91bedaae; no voters: 5dae8595b9484b57bc7ccea543f83130, de62034f80d9463dbd6386fc3e90fdaf
I20250625 08:09:51.678871 7817 raft_consensus.cc:2747] T 18ea19fdd2814cc08c6315b3463bd3a9 P eebfa8472fe04080ac012e2c91bedaae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 08:09:51.909767 7837 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5f0b114bd68344b0bd2ee322bd5eccdc)
I20250625 08:09:51.909904 7837 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0b114bd68344b0bd2ee322bd5eccdc" member_type: VOTER last_known_addr { host: "127.1.19.126" port: 36875 } } peers { permanent_uuid: "0404561252194ce29bf2924fd5e67e22" member_type: VOTER last_known_addr { host: "127.1.19.125" port: 38317 } } peers { permanent_uuid: "755093bec4b54a9588598f814c3708e1" member_type: VOTER last_known_addr { host: "127.1.19.124" port: 37745 } }
I20250625 08:09:51.910348 7837 leader_election.cc:290] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875), 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745)
W20250625 08:09:51.911470 3728 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 5f0b114bd68344b0bd2ee322bd5eccdc (127.1.19.126:36875): Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:51.911582 3726 leader_election.cc:336] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 755093bec4b54a9588598f814c3708e1 (127.1.19.124:37745): Network error: Client connection negotiation failed: client connection to 127.1.19.124:37745: connect: Connection refused (error 111)
I20250625 08:09:51.911643 3726 leader_election.cc:304] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0404561252194ce29bf2924fd5e67e22; no voters: 5f0b114bd68344b0bd2ee322bd5eccdc, 755093bec4b54a9588598f814c3708e1
I20250625 08:09:51.911768 7837 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0404561252194ce29bf2924fd5e67e22 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250625 08:09:54.521220 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer 5dae8595b9484b57bc7ccea543f83130 (127.1.19.67:37005): Couldn't send request to peer 5dae8595b9484b57bc7ccea543f83130. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 321: this message will repeat every 5th retry.
I20250625 08:09:55.440692 3709 master_runner.cc:186] Error getting master registration for 127.1.19.126:36875: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111),
I20250625 08:09:55.440816 3709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 08:09:55.570534 1647 consensus_peers.cc:487] T 99a2e60d7c7a4c419b609aa64d8b7334 P eebfa8472fe04080ac012e2c91bedaae -> Peer de62034f80d9463dbd6386fc3e90fdaf (127.1.19.66:46071): Couldn't send request to peer de62034f80d9463dbd6386fc3e90fdaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a2e60d7c7a4c419b609aa64d8b7334. This is attempt 321: this message will repeat every 5th retry.
I20250625 08:09:56.226974 3730 maintenance_manager.cc:419] P 0404561252194ce29bf2924fd5e67e22: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3217 bytes on disk
I20250625 08:09:56.229712 3729 maintenance_manager.cc:643] P 0404561252194ce29bf2924fd5e67e22: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.000s sys 0.002s Metrics: {"dirs.queue_time_us":168,"thread_start_us":167,"threads_started":2,"wal-append.queue_time_us":191}
W20250625 08:09:56.937389 2036 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:56.937623 2036 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:56.937680 2036 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
W20250625 08:09:56.937712 2036 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
F20250625 08:09:56.948288 2062 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1750838996 (unix time) try "date -d @1750838996" if you are using GNU date ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000044d) received by PID 1101 (TID 0x7ffbb33db700) from PID 1101; stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954149 2048 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954149 2048 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954172 2052 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_174089cfe43b4dbb9ed1b9c2921f9187 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7ffbd02b7980 (unknown) at ??:0
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954149 2048 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954172 2052 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_174089cfe43b4dbb9ed1b9c2921f9187 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956305 2065 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954149 2048 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954172 2052 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_174089cfe43b4dbb9ed1b9c2921f9187 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956305 2065 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956344 2061 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7ffbcdec9fb7 gsignal at ??:0
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954149 2048 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954172 2052 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_174089cfe43b4dbb9ed1b9c2921f9187 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956305 2065 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956344 2061 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957541 2043 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98189dd9ad454e5d8ee6edd848927c15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954149 2048 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954172 2052 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_174089cfe43b4dbb9ed1b9c2921f9187 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956305 2065 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956344 2061 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957541 2043 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98189dd9ad454e5d8ee6edd848927c15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957652 2049 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_31023e55a4384ac88f8415871e52d142 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954149 2048 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954172 2052 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_174089cfe43b4dbb9ed1b9c2921f9187 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956305 2065 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956344 2061 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957541 2043 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98189dd9ad454e5d8ee6edd848927c15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957652 2049 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_31023e55a4384ac88f8415871e52d142 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957787 2051 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d78071777f3646548f7f75cea674ecdc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954149 2048 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954172 2052 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_174089cfe43b4dbb9ed1b9c2921f9187 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956305 2065 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956344 2061 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957541 2043 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98189dd9ad454e5d8ee6edd848927c15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957652 2049 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_31023e55a4384ac88f8415871e52d142 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957787 2051 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d78071777f3646548f7f75cea674ecdc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957892 2063 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
F20250625 08:09:56.949347 2067 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950484 2068 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.950532 2057 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951661 2064 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951673 2059 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.951685 2050 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f22b18eed5c34ffaa7af847a3c64598d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952832 2058 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.952991 2056 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953060 2045 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_28db8798a1444ed2b1d46a5a3eba537b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953207 2053 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_128075ffc1db4314b52b7c28536f6e89 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953299 2055 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953400 2044 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_035cffaa28ec4a838bb218bb37230370 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.953490 2066 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954149 2048 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ba2c9866b9f44ecd85b09bccf0e63aa6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.954172 2052 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_174089cfe43b4dbb9ed1b9c2921f9187 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956305 2065 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.956344 2061 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 11*** Check failure stack trace: ***
1)F20250625 08:09:56.957541 2043 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_98189dd9ad454e5d8ee6edd848927c15 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957652 2049 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_31023e55a4384ac88f8415871e52d142 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957787 2051 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d78071777f3646548f7f75cea674ecdc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957892 2063 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)F20250625 08:09:56.957965 2046 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bd43623a5f9f40c58ad750234777ba9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.1.19.126:36875,127.1.19.125:38317,127.1.19.124:37745, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.1.19.126:36875: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7ffbcdecb921 abort at ??:0
@ 0x7ffbced32dcd google::LogMessage::Fail() at ??:0
@ 0x7ffbced36b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ffbced327cc google::LogMessage::Flush() at ??:0
@ 0x7ffbced33f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x55f79ef45432 kudu::MasterStressTest::DeleteTableThread() at ??:0
@ 0x55f79ef577b0 std::__invoke_impl<>() at ??:0
@ 0x55f79ef554ec std::__invoke<>() at ??:0
@ 0x55f79ef5cb3d _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x55f79ef5bb41 std::thread::_Invoker<>::operator()() at ??:0
@ 0x55f79ef5a8be std::thread::_State_impl<>::_M_run() at ??:0
@ 0x7ffbce5516df (unknown) at ??:0
@ 0x7ffbd02ac6db start_thread at ??:0
@ 0x7ffbcdfac71f clone at ??:0